Data quality methodology

The data quality methodology information is organized by analytical function and provides in-depth knowledge and best practices for your data quality strategy.

There are a number of key concepts with data quality analysis and monitoring that include capabilities to:

Support business-driven rule definition and organization
Apply rules and reuse consistently across data sources
Leverage multi-level rule analysis to understand broader data quality issues
Evaluate rules against defined benchmarks/thresholds
Assess and annotate data quality results
Monitor trends in data quality over time
Deploy rules across environments
Run data quality activities on either an impromptu or scheduled basis using either the user interface or the command line

Data quality analysis and monitoring
Data quality analysis and monitoring is a set of capabilities within IBM InfoSphere Information Analyzer designed to evaluate data against specific customer criteria. These rule evaluation criteria can be used repeatedly over time to see important changes in the quality of the data being validated.
Structuring data rules and rule sets
The data quality analysis capabilities of IBM InfoSphere Information Analyzer leverage some key concepts that are at the foundation of the ability to reuse data rules across multiple data sources and evaluate data rules in a larger set-based concept.
Naming standards
All rule and rule set definitions, all executable data rules and rule sets, all metrics, and all global variables require a name. A name can either facilitate or hinder reuse and sharing of these components, so while names can be freeform and be quickly created when simply evaluating and testing conditions, it is critical to identify an effective naming standard for ongoing quality monitoring.
Data rules analysis
At the heart of data rules analysis, the business wants to build data rules to test and evaluate for specific data conditions.
Data rule sets
IBM InfoSphere Information Analyzer also provides a mechanism where you can group multiple data rules that use the same source data into a data rule set. Based on all of the individual data rule results, the data rule set also produces overall statistics that apply to that source data in addition to the statistics resulting from the individual data rules. You can also establish benchmarks for data rule set results. When you run a data set rule, the actual results are compared against the benchmarks to determine the relevant variances, all of which can be monitored and trended.
Metrics
Metrics are user-defined objects that do not analyze data but provide mathematical calculation capabilities that can be performed on statistical results from data rules, data rule sets, and metrics themselves.
Monitoring results
Data rules, rule sets, and metrics are all executable objects that can be run as needed or scheduled (either internally or via a command line interface).
Deploying rules, rule sets, and metrics
Data rules, rule sets, and metrics in a data quality monitoring environment are typically targeted at production data, which requires more explicit control over tasks, such as definition and change.
Managing a data quality rules environment
As the users in your environment focus on more business areas and more systems, creating more rule definitions, and sharing information across IBM InfoSphere Information Analyzer projects, there is a greater need for management of this data quality environment.

Parent topic: Methodology and best practices