Data quality issues and challenges

Data quality issues are flaws in datasets that can compromise decision-making and other data-driven workflows at an organization. Common examples include duplicate data, inconsistent data, incomplete data and data silos.

While data quality issues have long dogged data analysis, addressing them has arguably taken on unprecedented significance in the modern, big data era. Enterprises increasingly rely on large, complex datasets to unlock key insights and gain a competitive advantage.

But any data quality issues within these datasets degrade the accuracy of analysis and business intelligence. This “bad data” results in missed opportunities, inefficiencies in business processes, financial losses and regulatory penalties.

The impact of poor data quality also extends to artificial intelligence (AI) initiatives: Machine learning algorithms must be paired with high-quality datasets to produce performant machine learning models.

Without good training data, resulting models are more likely to make inaccurate, irrelevant predictions, imperiling AI-powered initiatives. Gartner predicts that “through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.”

Data quality issues stem from a number of factors, including human error and data integration problems. Practices such as data audits, data observability and data governance can elevate the quality and usability of an organization’s data, improving business decisions and business outcomes.

While different organizations face different data quality problems, here are among the most common data quality issues:

Inaccurate data refers to data points that fail to represent real-world values, diminishing the overall quality of the dataset that houses them. Inaccurate data not only hinders decision-making but can also prevent organizations from successfully implementing innovative tools, including AI-powered solutions.

For instance, inaccurate data on retail worker shifts led to poor performance for an AI scheduling tool used at over 6,000 stores. Researchers found that managers chose to manually override scheduling of 84% of the shifts in the AI-generated timetables.¹

Data accuracy has also emerged as a key concern for organizations considering adopting agentic AI. Nearly half (49%) of executives cited data inaccuracies and bias as a barrier to embracing the technology, according to the IBM Institute for Business Value.²

Data duplication occurs when data is replicated either purposely or inadvertently. In the case of the former, duplication can provide a safeguard in case of data loss: When data in one location is damaged or goes missing, systems can continue to function using copies of the data stored elsewhere.

However, unintentional duplicate data entries can cause a variety of problems. They can over-represent specific data points or trends, resulting in unreliable outputs and skewed forecasts. Duplicate data can also cause increased storage costs and slower performance. Unintentional duplication can occur because of flaws in data migration and integration processes—if, for instance, two different data sources contain the same record, both copies might remain when data from those sources is combined in a single dataset.

Inconsistent data creates discrepancies in the representation of real-world situations. For example, the number of employees in a company department should not exceed the total number of employees at that business.

Data can also be inconsistent when it represents the same types of values in different formats. For example, a dataset of street names can store full street names (Jones Street) or abbreviated names (Jones St.) but should not include both.

Data assets fall short on the data completeness dimension when tables are missing values or entire rows. Incomplete or empty data values can interrupt data integration processes and take up memory on source systems.

Incomplete data can sometimes lead researchers to delete records, even when those records include information (missing values notwithstanding) that could provide valuable contributions to their data analysis.³

And as governments increasingly require greater transparency in regulated industries, incomplete data can result in costly penalties. Banking giant JPMorgan Chase was fined roughly USD 350 million by US banking regulators in 2024 for providing incomplete trading and order data to surveillance platforms.⁴

Invalid data does not fall within a range of permitted values or or violates data type, format or business rules. For example, if a customer data table includes a column for ages 0 to 125, an entry of 200 would be invalid.

There is an overlap between invalid data issues and incomplete data issues: When a mandatory data field is left blank, that is also a case of invalid data. In addition, a table may be considered invalid if its data is not organized according to a specified schema.

In an era where information delays and lags increasingly yield negative consequences, data freshness and timeliness are critical for smart decision-making. When data is not regularly updated, the decline of data quality over time is known as data decay.

Outdated information can produce outcomes that don’t serve present-day circumstances. One study found that 85% of companies blamed stale data for bad decision-making and lost revenue.⁵

Data labeling is the process of identifying raw data—such as images, text files or videos—and assigning one or more labels to specify its context for machine learning models. These labels help the models interpret the data for better predictions.

However, large-scale datasets compiled from web scraping or crowdsourcing platforms are often mislabeled. This phenomenon, known as label noise, can reduce the accuracy of model predictions.⁶

Biased data is data skewed by one or more human biases, such as cognitive bias, confirmation bias, historical bias and sampling bias. Data bias has emerged as a major quality issue for AI model training, contributing to inaccurate outputs that can result in a host of consequences, including legal liability, discrimination and poor customer service.

For example, biased data may have undermined patient care during the Covid-19 pandemic. AI-powered analysis on data from pulse oximeters (devices that measure blood oxygen saturation) helped inform treatment decisions.

The devices’ sensors, however, worked less effectively on people with darker skin, thicker skin or smaller fingers—raising concerns about the reliability of the analysis.⁷

Data silos are isolated collections of data that prevent data sharing among systems and business units. This isolation can precipitate other data issues, such as inconsistency.

Silos can also prevent organizations from leveraging relevant data for specific use cases. For instance, a marketer planning customer outreach would benefit from a database that contains customer names, contact information and records of previous outreach attempts, rather than a database of names and contact information alone.

Data quality issues are an age-old problem. In the mid 1800s, British nursing pioneer and healthcare reformer Florence Nightingale described military medical statistics as rife with so many inaccuracies that they existed in “a state of great confusion, so that it is hardly possible to obtain correct results.” ⁸

In Nightingale’s time, human error and biases would have caused the bulk of data quality issues. But today the landscape of causes is more complex, with risks emerging at multiple points of a data pipeline.

Organizations can prevent and address data quality problems through data quality management, data quality monitoring and data quality tools that support these practices. Key measures include:

• Implementing data governance
• Detecting issues
• Correcting errors
• Validating data
• Monitoring data 

Through the discipline of data governance, organizations set policies and standards necessary for collecting, storing and maintaining high-quality data.

Data governance software solutions support the enforcement of these policies by making it easy for users to discover, understand and access data. Leading solutions include searchable catalogs of data assets, as well as metadata enrichment and quality check capabilities—features that ensure that metadata includes information on data rules, data definitions and data lineage.

Data profiling and auditing tools evaluate the structure and context of data, and establish a baseline against which to measure remediation. The right tool can identify issues such as inconsistencies, duplicate records and general anomalies.

Data cleansing (also known as data cleaning) is the correction of errors and inconsistencies in raw datasets through methods such as standardization, data deduplication and addressing missing values.

While data engineers and other data management professionals can perform data cleaning techniques through manual methods, AI can be used to automate and optimize data cleaning processes such as standardizing data and consolidating duplicates.

Data validation is rule-based verification that data is clean, accurate and meets specific data quality requirements (such as range constraints) that make it ready for use. Data validation capabilities are commonly included in data integration platforms.

While it’s critical for organizations to ensure the quality of data entering their systems, it’s also important to continue monitoring it throughout its lifecycle.

Leading data observability tools allow the assessment of data across an organization’s data ecosystem and the management of incidents through a single dashboard. Such tools enable automated monitoring, root cause analysis, logging, data lineage, service level agreement (SLA) tracking and the routing of real-time data anomaly alerts to data stakeholders.