What is data validation?

Data validation is the process of verifying that data is clean, accurate and ready for use.
 

Valid data falls within permitted limits or ranges, conforms to specified data formats, is free of inaccuracies and adheres to an organization’s own specific validation criteria.

Data validation is a long-established step in data management workflows—invalid data, after all, can wreak havoc on data analysis. However, its urgency and significance has multiplied as organizations engage in unprecedented levels of data collection to power data-driven decision-making and artificial intelligence (AI) initiatives.

Today, enterprises routinely amass large datasets containing terabytes or petabytes of data. This information comes from various data sources, such as Internet of Things (IoT) devices or social media, and is often moved to data warehouses and other target systems. But information coming from a wide range of sources, combined with the scale of massive data migrations, can set the stage for a host of problems: inconsistent formats and discrepancies, duplicate data, incomplete data fields, data entry errors and even data poisoning.

These data quality problems can compromise data integrity and imperil informed decision-making. And invalid data doesn’t only create headaches for data analysts; it’s also a problem for engineers, data scientists and others who work with AI models.

AI models, including machine learning models and generative AI models, require reliable, accurate data for model training and performance. As effective AI implementation becomes a critical competitive advantage, businesses can’t afford to have invalid data jeopardize their AI efforts. Enterprises use data validation processes to help ensure the quality of data is sufficient for use in data analytics and AI.

In addition, data validation has become increasingly important in relation to regulatory compliance. For instance, the EU Artificial Intelligence Act requires that data validation for “high-risk” AI systems be subject to rigorous data governance practices.

Data validation entails the establishment and enforcement of business rules and data validation checks.

While different organizations use different rules and data validation techniques, the most common types of data validation checks are:

• Code checks
• Consistency checks
• Data type checks
• Format checks
• Range checks
• Uniqueness checks

A code check determines whether a data value is valid by comparing it to a list of acceptable values. Some examples include country codes, International Standard Book Number (ISBN) codes and North American Industry Classification System (NAICS) codes for classifying businesses.

Consistency checks confirm that input data is logical and does not conflict with other values. For instance, in a database of married couples, the dates of their engagements should be earlier than their wedding dates.

A data type defines the valid format for data in a particular column. Examples of data types might include text, numeric or date. This check identifies each value that does not match the selected data type in length, precision or scale—or that violates the specified data type.

Format checks are implemented for columns that have specific data formatting requirements, such as columns for phone numbers, email addresses and dates.

Range checks determine whether numerical data falls within a predefined range of minimum and maximum values. For example, a column of acceptable vehicle tire pressures might range from 30 to 35 pounds per square inch.

Uniqueness checks apply to columns where every data entry must be unique and there are no duplicate values.

Other checks used for data validation include length checks (ensuring the right number of characters in a field); presence checks (ensuring mandatory fields are not blank); and schema validation (ensuring that data conforms to a predefined structure).

Data validation is often mentioned in the same breath as data cleansing, which is the correction of errors and inconsistencies in raw datasets. Sometimes data validation is considered a component of data cleansing, while in other cases it is referred to as a distinct process.

Both data validation and data cleansing are elements of data quality management (DQM), a collection of practices for maintaining high-quality data at an organization. Additional, complementary DQM processes include data profiling, data quality monitoring and metadata management.

While data validation can be conducted manually, it can be an arduous and time-consuming task. Different data tools can help data professionals accelerate, automate and streamline the data validation process.