Cloud Data Lake vs. Data Warehouse vs. Data Mart

Three distinct types of cloud storage repositories exist today, each serving a different purpose to address a specific need:

Data lake

A data lake is a large repository of raw data, either unstructured or semi-structured. This data is aggregated from various sources and is simply stored. It is not altered to suit a specific purpose or fit into a particular format. To prepare this data for analysis involves time-consuming data preparation, cleansing and reformatting for uniformity. Data lakes are great resources for municipalities or other organizations that store information related to outages, traffic, crime or demographics. The data could be used at a later date to update DPW or emergency services budgets and resources.

Data warehouse

A data warehouse is an aggregation of data from many sources to a single, centralized repository that unifies the data qualities and format, making it useful for data scientists to use in data mining, artificial intelligence (AI), machine learning and, ultimately, business analytics and business intelligence. Data warehousing could be used by a large city to aggregate electronic transactions from various departments, including speeding tickets, dog licenses, excise tax payments and other transactions. This structured data would be analyzed by the city to issue follow-up invoicing and to update census data and police logs. It could also be used by a developer to aggregate terabytes of data generated by sensors on automobiles to aid in the decision-making process for an autonomous driving solution.

Data mart

A data mart is a subset of a data warehouse that benefits a specific set of users within the business or business unit. A data mart could be used by the marketing department of a manufacturing company to determine the ideal target demographic or persona to aid in the development of marketing plans. It could also be used by a manufacturing department to analyze performance and error rates to enable continuous improvement. Data sets within a data mart are often utilized in real time, for current analysis and actionable results.

While all three types of cloud data repositories hold data, there are very distinct differences between them. For instance, a data warehouse and a data lake are both large aggregations of data, but a data lake is typically more cost-effective to implement and maintain because it is largely unstructured. 

Data lake architecture has evolved over the past few years to support larger volumes of data and cloud-based computing. Large amounts of data are received from a number of data sources to a central location. 

A data warehouse could be structured in one of three ways:

1. As a managed service offered by cloud providers.
2. As a software solution that provides in-house control and strict security protocols, which can be helpful when dealing with regulation compliance.
3. As an appliance, which is usually a plug-and-play bundled software and hardware solution.

Data within a data warehouse can be more easily utilized for various purposes than data within a data lake. The reason is because a data warehouse is structured and can be more easily mined or analyzed.

A data mart, on the other hand, contains a smaller amount of data as compared to both a data lake and a data warehouse, and the data is categorized for a specific use or by a specific demographic or business unit. A data mart can exist in many different formats (star, snowflake or vault) defined by the logical structure of the data, with a vault structure being more agile, flexible and scalable than the other formats.

There are three types of data marts:

1. A dependent data mart, which consists of enterprise data warehouse partitions. It is a subset of primary data in a warehouse.
2. An independent data mart, which is a standalone system, siloed to a specific part of the business.
3. A hybrid data mart, which consists of data from a warehouse and independent sources. This type typically provides faster data access and a user-friendly interface.

The type of data repository you choose, and the structure of it, is highly dependent on the needs and demands of your business. If it makes sense for your business, take advantage of the benefit of hybrid cloud-based storage for flexibility, scalability and a broader, informed approach to problem-solving and decision-making.

Manufacturing

A large multinational manufacturing company generates large volumes of data for various uses. Some of the data is important, while other data may or may not have a purpose in the future. The company uses a cloud-based data warehouse for storage of bulk data, which is less expensive than other data storage options. However, the company also has dependent data marts in place for specific areas of the business, providing value to business users in departments like finance, manufacturing and marketing. Each of these marts contains data earmarked for a specific use, formatted to make it easy to analyze. For example:

• The finance department uses its data mart to prepare customer account statements and maintain balance sheets.
• The manufacturing department uses its data mart to analyze assembly line efficiency, process data to input into AI solutions and maintain procurement databases.
• The marketing department uses its data mart to determine the effectiveness of campaigns and communication while analyzing and collating survey responses.

Large municipality

A large municipality needs an affordable solution that provides data in an affordable and somewhat usable manner. The municipality uses a data lake in the cloud to maintain traffic data. It can’t afford to analyze and take action on that data at the moment but will be ready to when funding comes through. It also uses a software data warehouse on-premises to track tax bill status. In addition, the municipality uses a hybrid data mart to track the spread of a virus among residents, aggregating data from various hospitals and municipal health services to a single repository to be analyzed and used by the department of health.

There are many misconceptions regarding cloud-based data repositories. Some of the most common misconceptions include the following:

• One size fits all: This is absolutely not the case when considering cloud data storage solutions. Each business has different budgetary constraints, goals, resource allocations and preferences. It is important to evaluate your business needs and budget and let that dictate the solution that will help you achieve your goals.
• Data islands leave your data stranded in a repository: This is false. The very nature of cloud-based storage is that it allows access to the data from anywhere, with proper permissions.
• Cloud-based solutions are less secure:  In actuality, cloud providers can offer stronger security, providing regular updates and the most current protocols available. They often have teams of security experts with the most current certifications dedicated to ensuring the most stringent security solution is protecting your data. Many providers also have teams working with regulatory compliance bodies to optimize their solution. However, in some industries (such as healthcare and finance), regulatory compliance could require the ability to access data without an Internet connection, which would require on-premises equipment.
• Cloud-based data repositories are expensive: Cloud-based storage can be less expensive than on-premises solutions because there are no large up-front infrastructure investments, cooling or floor-space costs, ongoing maintenance costs or teams of in-house experts required. Monthly costs vary by vendor or cloud provider.

Your business is unique, with specific resources, goals, and challenges. Evaluate your options carefully to determine what solution will best serve your needs. Consider the following:

• Your business and technology goals
• Your budget
• The volume of data in need of storage
• How frequently you will need to access it
• Whether you have specific needs today or in the short term

These considerations will help you determine what solution, or combination of solutions, will help you reach your goals.

IBM offers several solutions to assist with your cloud storage and data science needs.

• IBM Db2 Warehouse on Cloud is an elastic cloud data warehouse that offers independent scaling of storage and compute. Smaller data marts can use the Flex One feature, which is an elastic data warehouse built for high-performance analytics. This system is deployable on multiple cloud providers, starting at 40 GB of storage.
• Another option worth considering is IBM InfoSphere® Master Data Management (MDM). This customizable system manages all aspects of your critical enterprise data, giving users access in a single-trusted view. Through this streamlined dashboard, users are empowered to conduct detailed analysis, gain actionable insight, and ensure total compliance with data governance and policies across the entire enterprise.
• Netezza Performance Server, the next evolution of the IBM Netezza appliance, builds on the hyper-converged architecture of the IBM Cloud Pak for Data System to provide a cloud native decision support system for your enterprise’s most complex analytics. It is also available now on AWS and Azure.
• IBM Watson Studio, a data-science and machine-learning offering, empowers organizations to tap into data assets and inject predictions into business processes and modern applications.