What is data quality management?

Data quality management, or DQM, is a collection of practices for enhancing and maintaining the quality of an organization’s data.

As the global production of data continues at a breathtaking pace, effective data quality management helps enterprises avoid low-quality data, which can lead to costly errors and inefficiencies in business processes. With trusted, reliable data at their fingertips, enterprises can unlock valuable insights, achieve better decision-making and integrate artificial intelligence (AI) into their business operations.

Data quality management includes practices such as data profiling, data cleansing, data validation, data quality monitoring and metadata management. Successful data quality management results in datasets optimized for key dimensions of quality such as accuracy, completeness, consistency, timeliness, uniqueness and validity.

Software solutions can help organizations and data practitioners address data quality issues and create high-quality data pipelines. These tools offer features such as data quality analysis, automated anomaly detection, real-time incident alerts and more.

To understand the importance of data quality management, consider what can happen in its absence: As enterprises prioritize data-driven functions, poor data quality can result in errors, delays, financial losses and reputational damage, among other serious consequences. Such risks are multiplied in the era of “big data,” as organizations grapple with massive and complex datasets.

Imagine the following “bad data” scenarios:

• A retailer’s customer data table is riddled with inaccuracies, giving rise to misdirected and ineffective marketing strategies.
  
  
• A clinical study contains inconsistent formats, making it difficult to compare data elements and hindering research on disease progression and healthcare.
  
  
• A business in a highly regulated industry is plagued by data quality problems, running afoul of government laws and regulations such as GDPR or the Sarbanes-Oxley (SOX) Act.

In contrast, high-quality data contributes to business intelligence initiatives, yielding operational efficiency, optimized workflows, regulatory compliance, customer satisfaction and enterprise growth.

The benefits of high data quality have further intensified with the widespread adoption of artificial intelligence. Algorithms require high-quality data for effective model performance; good data quality can enable more precise and useful AI model outputs.

In fact, enterprises with large stores of data trusted by internal and external stakeholders realized nearly double the return on investment on their AI capabilities, according to research by the IBM Institute for Business Value.

Successful data quality management ensures that an organization’s data meets six key data quality dimensions:

• Accuracy
• Completeness
• Consistency
• Timeliness
• Uniqueness
• Validity

While these are among the most common data quality dimensions used by data practitioners, other data quality metrics include accessibility, relevance, concise representation and appropriate amount of data or volume.¹

Common, complementary data quality management practices among data stewards and other data professionals include:

• Data profiling
• Data cleansing
• Data validation
• Data quality monitoring
• Metadata management

Before improving data, it’s important to determine where improvement is needed. Data profiling is the process of reviewing the structure and content of existing data to evaluate its quality and establish a baseline against which to measure remediation.

Analysis conducted during data profiling can provide information on data types, reveal anomalies, identify invalid or incomplete data values and assess relationships between datasets.

Data cleansing, also known as data cleaning, is the correction of errors and inconsistencies in raw datasets. Methods for achieving clean data include standardization (making formats and structures consistent), adjusting or removing outliers, data deduplication and addressing missing values.

Sometimes considered part of data cleansing approaches, data validation is the verification that data is clean, accurate and meets specific data quality rules and requirements (such as range or referential integrity constraints) that make it ready for use.

Ensuring data quality is an ongoing process. Schema changes, data staleness and duplicate records can all compromise data integrity over time. Continuous data monitoring identifies existing data assets that no longer meet an organization’s data quality standards and key performance indicators (KPIs).

While metadata management supports multiple capabilities, such as security and governance, it is also often included under the umbrella of DQM. Metadata management techniques such as metadata enrichment can ensure that metadata includes information on data rules, data definitions and data lineage. This can inform and streamline data management efforts, including data quality initiatives.

Data quality management, data management, master data management and data governance are distinct but related processes for optimizing the value of an organization’s data assets.

Data management encompasses the oversight and handling of data throughout its lifecycle. Data management strategies help organizations address the use of diverse data sources and plan for data disaster recovery, among other issues. Data quality management can be considered a discipline or subset of data management.

Master data management is a comprehensive approach that establishes consistency for the handling of critical data (master data) across an organization.

Through master data management, critical data is shared and used by various applications and systems within the organization to reduce data fragmentation, siloed data, duplication and inaccuracies. It does so through a collection of processes and technological tools, some of which are also incorporated into data quality management, such as data cleansing.

Data governance defines and implements policies, standards and procedures for data collection, data storage, ownership, processing and use. Like data quality management, data governance can also be considered a data management discipline. At the same time, the procedures established through data governance frameworks, such as governance policies on the consistent handling of data, can support DQM initiatives.

Data quality management tools and software solutions can significantly reduce manual DQM efforts. And while the proliferation of AI is one of the driving factors behind the need for data quality management, AI also enables more powerful DQM solutions. Machine learning, for instance, can be deployed for automated data anomaly detection.

Other capabilities offered by data quality management solutions include:

• Predefined data quality checks and customizable rules
  
  
• Data catalogs with built-in data quality analysis
  
  
• Comprehensive dashboards for data incident management
  
  
• Real-time alerts for anomalies and other data issues
  
  
• Root cause analysis to inform incident resolution
  
  
• Metadata lineage tracking for transparency in data transformation