Poor data quality is one of the top barriers faced by organizations aspiring to be more data-driven. Ill-timed business decisions and misinformed business processes, missed revenue opportunities, failed business initiatives and complex data systems can all stem from data quality issues. Just one of these problems can prove costly to an organization. Having to deal with all of them can be devastating.
Several factors determine the quality of your enterprise data like accuracy, completeness, consistency, to name a few. But there’s another factor of data quality that doesn’t get the recognition it deserves: your data architecture.
How the right data architecture improves data quality
The right data architecture can help your organization improve data quality because it provides the framework that determines how data is collected, transported, stored, secured, used and shared for business intelligence and data science use cases.
The first generation of data architectures represented by enterprise data warehouse and business intelligence platforms were characterized by thousands of ETL jobs, tables, and reports that only a small group of specialized data engineers understood, resulting in an under-realized positive impact on the business. Next generation of big data platforms and long running batch jobs operated by a central team of data engineers have often led to data lake swamps.
Both approaches were typically monolithic and centralized architectures organized around mechanical functions of data ingestion, processing, cleansing, aggregation, and serving. This created number of organizational and technological bottlenecks prohibiting data integration and scale along several dimensions: constant change of data landscape, proliferation of data sources and data consumers, diversity of transformation and data processing that use cases require, and speed of response to change.
What does a modern data architecture do for your business?
A modern data architecture like Data Mesh and Data Fabric aims to easily connect new data sources and accelerate development of use case specific data pipelines across on-premises, hybrid and multicloud environments. Combined with effective data lifecycle management, which evolves into data as product management, a modern data architecture can enable your organization to:
Allow data stewards to ensure data compliance, protection and security
Enhance trust in data by getting visibility into where data came from, how it has changed, and who is using it
Monitor and identify data quality issues closer to the source to mitigate the potential impact on downstream processes or workloads
Efficiently adopt data platforms and new technologies for effective data management
Apply metadata to contextualize existing and new data to make it searchable and discoverable
Perform data profiling (the process of examining, analyzing and creating summaries of datasets)
Reduce data duplication and fragmentation
Because your data architecture dictates how your data assets and data management resources are structured, it plays a critical role in how effective your organization is at performing these tasks. Meaning, data architecture is a foundational element of your business strategy for higher data quality. Critical capabilities of modern high-quality data quality management solutions require an organization to:
Perform data quality monitoring based on pre-configured rules
Build data modeling lineage to perform root cause analysis of data quality issues
Make a dataset’s value immediately understandable
Practice proper data hygiene across interfaces
How to build a data architecture that improves data quality
A data strategy can help data architects create and implement a data architecture that improves data quality. Steps for developing an effective data strategy include:
1. Outlining business objectives you want your data to help you accomplish
For example, a financial institution may look to improve regulatory compliance, lower costs, and increase revenues. Stakeholders can identify business use cases for certain data types, such as running data analytics on real-time data as it’s ingested to automate decision-making to drive cost reduction.
2. Taking an inventory of existing data assets and mapping current data flows
This step includes identifying and cataloging all data throughout the organization into a centralized or federated inventory list, thereby removing data silos. The list should detail where each dataset resides and what applications and use cases rely on it. Next, select the data needed for your key use cases and prioritize those data domains that included it.
3. Developing a standardized nomenclature
A naming convention and aligned data format (data classes) for data used throughout the organization helps to ensure data consistency and interoperability across departments (domains) and use cases.
4. Determining what changes must be made to the existing architecture
Decide on the changes that will optimize your data for achieving your business objectives. Researching the different types of modern data architectures, such as a data fabric and data mesh can help you decide on the data structure most suitable to your business requirements.
5. Deciding on KPIs to gauge a data architecture’s effectiveness
Create KPIs and use advanced analytics that link the measure of your architecture’s success to how well it supports data quality.
6. Creating a data architecture roadmap
Companies can develop a rollout plan for implementing data architecture and governance in three to four data domains per quarter.
Data architecture and IBM
A well-designed data architecture creates a foundation for data quality through transparency and standardization that frames how your organization views, uses and talks about data.
As previously mentioned, a data fabric is one such architecture. A data fabric automates data discovery, governance and data quality management and simplifies self-service data access to data distributed across a hybrid cloud landscape. It can encompass the applications that generate and use data, as well as any number of data storage repositories such as data warehouses, data lakes (which store vast amounts of big data), NoSQL databases (which store unstructured data) and relational databases that utilize SQL.