Automating data quality remediation

How data quality remediation works

Data quality issues can be time-consuming to investigate and fix. The data quality remediation capability helps speed up that process:

1. Analyze violations automatically: When a data quality check fails, watsonx.data intelligence analyzes what went wrong and why.
2. Create detailed tickets: Watsonx.data intelligence generates tickets in GitHub that include the issue context, root cause analysis, and suggestions for fixing the problem.
3. Work with AI assistants: AI assistants like IBM Bob can read these tickets and help you fix the issues by generating code, suggesting changes, or guiding you through the remediation steps.

Requirements and restrictions

The following requirements and restrictions exist.

Cloud platforms

IBM Cloud AWS

Some functionality might not be available in all regions on all cloud platforms. See Regional availability.

Required services or service plans

DataStage must be provisioned in addition to watsonx.data intelligence.

Required permissions

To enforce data contracts through APIs, you must have the Admin or the Editor role in the project and the Manage data quality assets and Execute data quality rules user permissions.

For managing data contracts in a data product, you must have the following roles:

• IBM watsonx role: Manager or Data Product Hub Administrator
• Data Product Hub collaborator role: Admin

Required credentials

Task credentials are required for data contract testing. For more information, see Managing task credentials.

Supported data contract formats

YAML

Supported tracking tools

GitHub Enterprise

Supported AI assistants

IBM Bob, Claude, or other customer-selected AI assistants that can consume remediation tickets through standard interfaces

Configuring data quality remediation

To enable the remediation capabilities, complete the following steps:

1. Set up the tracking system for remediation tickets.

   1. In your GitHub Enterprise system, create a repository for storing, managing, and tracking remediation tickets.
   2. For this repository, create a personal access token with write permissions for issues. You can create a fine-grained or a classic personal access token. Select the scope repo with full control of private repositories.

2. Create project for data contract testing and configure the Git connection.

   1. Create a GitHub API connection to your GitHub Enterprise server. Select Bearer token as the authentication method and provide the personal access token that you created in the previous step.
   2. Go to the project's Manage tab and open the data quality settings.
   3. In the Agentic data quality remediation section, select the Git connection that you created and enter the name of the GitHub repository that you created earlier for tracking data quality issues.

3. Configure remediation behavior. Add the following custom properties to your data contract either in the YAML file or in the data product configuration UI.

   remediationRequired: true

   Required property to enable generation of remediation tickets.

   remediationInstruction: "<instructions-for-remediation>"

   Optional property to provide instructions for ticket generation. If you don't provide this property, the default instructions are used:

   Get the explanation for data contract violation. After retrieving the explanation, create a git issue using received title data and the obtained explanation.
   

   • In the data contract YAML file, add the entries to the global customProperties: section:

     - property: remediationRequired
       value: true
     - property: remediationInstruction
       value: "<instructions-for-remediation>"
     

     Additionally, you can provide id: and description: entries for these properties.

   • In the data product configuration UI, add the entries on the Custom properties page of the data contract.

4. In the data contract Tests configuration page in Data Product Hub, select the project where you configured the Git repository as the project for running data contract tests. In the API call for testing the data contract, use the equivalent project ID.

When data quality violations are detected during data contract testing, a ticket with the default title [ODCS Data Quality Violation] Contract Name: <contract_name> (Created At: <timestamp>) is created in the tracking system that you can evaluate and act upon.

Such a remediation ticket contains issue context, root-cause insights, and remediation guidance that external AI assistants can use to accelerate resolution.

However, if multiple data quality rules are executed in a single test run, not all error statuses might be included in the ticket, with higher-priority issues being reported first.

Each test run generates a new ticket where the explanations might differ from run to run even if the rules and the evaluated data did not change. For more information, see Accuracy considerations.

GET /data_quality/v4/projects/{project_id}/remediations/logs

Working with AI assistants

AI assistants like IBM Bob can read the detailed tickets and help you fix the issues. In general, such a ticket includes this information:

• What data quality rule failed
• Why it failed (root cause)
• Suggestions for fixing it
• Context about the data and contract

You can ask the AI assistant to:

• Explain the problem in simpler terms
• Generate code to fix the issue
• Suggest changes to your data quality rules
• Create a pipeline to prevent similar problems

You review and approve any changes before they're applied.

Learn more

• Ensuring data quality through data contracts
• Data Product Hub: Managing Data Contracts