Migrating data protection rules (IBM Knowledge Catalog)

Use API calls or cpd-cli commands to export and import data protection rules within a Cloud Pak for Data cluster or between different clusters.

You can export and import data protection rules with APIs. Alternatively, you can use the integrated Cloud Pak for Data command-line interface (cpd-cli) to export data protection rules from any system and import the rules into the same or different system. Links are maintained to glossary artifacts, catalogs, assets, and users. This task is also handy for backing up and restoring your data protection rules on the same or different system.

Prerequisites

Choose either API or cpd-cli commands:

Learn more

Prerequisites

Required permission

To complete this task, you must be an administrator of the project (namespace) where you deployed the IBM Knowledge Catalog service.

Before you begin

  • All artifacts, users, user groups, user roles, and assets that are used in rules must be imported correctly to the target system before the exported rules are imported.

  • To avoid any errors and changes to the enforcement behavior, you must verify that the Manage rule settings between the source and target systems are configured the same.

  • Before you import a rule, verify whether you have custom predicates in the exported rules. If so, you must create these custom predicates in the target system before you can import.

Limitations

The unmasked original data can persist in temporary storage for the following scenarios:

  • A cache of data to display in the Example data section when a data protection rule is created or edit.
  • A masked copy of the data in Cloud Object Storage for rapid recall of the masked data.
  • A profile data might not store the entire data but store an aggregation of the data, which can include values that are found in the data.

Migrating data protection rules by using API

Export

To export from one installation to another, set external_export to true by running the following API:

curl -k -X GET -H 'Content-Type: application/json' -H "Authorization:Bearer $token" '$url/v3/enforcement/rules/export?external_export=true'

An exported_file.json file is generated that can be used for import.

Import

  1. Before you import between different installations, generate a report of all rules that succeed or fail in matching names to IDs in the target system. By running the following API that sets the commit parameter to false:
curl -k -X POST -H 'Content-Type: application/octet-stream' -H "Authorization:Bearer $token" '$url/v3/enforcement/rules/import?external_import=true&commit=false' -d "@./exported_file.json"

Before you commit the rules, use the following report to identify any rules that fails to match IDs in the target system. When you run the import with external_import set to true, no rules are created and committed when the commit parameter is set to false:

{
  "failedList": [
    {
      "name": "export name4",
      "index": 4,
      "ruleGuidInRequest": "abcd123e-123b-1ab2-a123-ab123456c7d1",
      "reason": "WDPPS9040E: Import rules failed to match export component ids: [id=123a4b56-1a23-1234-a1bc-12a3b45cd6ef_123a4b56-1a23-1a23-1a23-12ab345cd67e]"
    }
  ],
  "matchedList": [
    {
      "name": "export name1",
      "index": 1,
      "ruleGuidInRequest": "12abc456-1234-12a3-a1bc-a1abc2345abc"
    },
    {
      "name": "export name2",
      "index": 2,
      "ruleGuidInRequest": "123456a7-1abc-1a2b-a1bc-ab1cd234ef56"
    },
    {
      "name": "export name3",
      "index": 3,
      "ruleGuidInRequest": "a123bcd4-a1b2-1234-1234-a1234567b123"
    }
  ]
}
  1. Run the following API to set the commit parameter to true, such that all or none of the rules can be committed. If one of the rules fails, the whole import fails:
curl -k -X POST -H 'Content-Type: application/octet-stream' -H "Authorization:Bearer $token" '$url/v3/enforcement/rules/import?external_import=true&commit=true' -d "@./exported_file.json"

A sample response when importing the export package (exported_file.json file) is successful:

SUCCEEDED
{
  "importedList": [
    {
      "name": "export name1",
      "index": 1,
      "guid": "a1b2cd3e-4567-8f90-gh12-345i6jklm78n",
      "ruleGuidInRequest": "12abc345-6789-01d2-e3fg-h4ijk5678lmn"
    },
    {
      "name": "export name2",
      "index": 2,
      "guid": "12a234b5-c678-90d1-ef2g-34567h8i9012",
      "ruleGuidInRequest": "123456a7-8bcd-9e0f-g1hi-jk2lm345no67"
    },
    {
      "name": "export name3",
      "index": 3,
      "guid": "1a234b5c-d6e7-89f0-1234-5g6hi7j8901k",
      "ruleGuidInRequest": "aa123bcd4-e5f6-7890-1234-g5678901h234"
    },
    {
      "name": "export name4",
      "index": 4,
      "guid": "12a345bc-6789-0d12-3ef4-g5h6ijk789lm",
      "ruleGuidInRequest": "abcd123e-456f-7gh8-i901-jk234567l8m9"
    }
  ]
}

A sample response of when importing the export package (exported_file.json file) fails:

FAILED
{
  "trace": "a1bc2d3e4fg5h6ijklmn7o9pq",
  "errors": [
    {
      "code": "WDPPS9043E",
      "message": "WDPPS9043E: Import rules failed: {\"failedList\":[{\"name\":\"export name4\",\"index\":4,\"ruleGuidInRequest\":\"abcd123e-123b-1ab2-a123-ab123456c7d1\",\"reason\":\"WDPPS9040E: Import rules failed to match export component ids: [id=123a4b56-7c89-0123-d4ef-56g7h89ij0kl_123m4n56-7o89-0p12-3q45-67rs890tu12v]\"}]}"
    }
  ]
}

Migrating data protection rules by using cpd-cli commands

Before you begin

  1. The cpd-cli command-line interface and related modules must be installed. See Installing the Cloud Pak for Data command-line interface (cpd-cli).
  2. Verify that you created a cpd-cli profile on your workstation. See Creating a profile to use the cpd-cli management commands.
  3. After you obtain the API key, complete the profile setup by running the following commands:
cpd-cli config users set admin --username <username> --apikey <apikey>
cpd-cli config profiles set default --user admin --url <url>

Export

To export data protection rules from the IBM Knowledge Catalog service, complete the following steps:

  1. Prepare a YAML file with the export specification. Create an export.yaml file with the following parameters:
  • parameter: EXPORT_HOST
    value: The hostname of the cluster from which you want to export data.
  • parameter: EXTERNAL_EXPORT
    value: true or false. This parameter is required. To export between different systems, set this parameter to true so that the IDs in rules from source system are converted to names at export.
  • parameter: AUTH_STRING
    value: The access token for the source system. You can specify either a bearer or ZenApiKey token. For more information about generating a ZenApiKey token, see Generating an API authorization token. For more information about generating a bearer token, see Generating a bearer token.

Make sure the key and indentation are as in the following example:

policy-api-aux:
    EXPORT_HOST: <hostname>
    EXTERNAL_EXPORT: true
    AUTH_STRING: Bearer <token> | ZenApiKey <token>

If the export.yaml file is not provided, the following default values are used during export:

  • EXPORT_HOST: <hostname> - current host
  • EXTERNAL_EXPORT: true
  • AUTH_STRING: - The token is obtained by the cpd-cli for the user defined in profiles specified by the required --profile parameter.
  1. Go to the directory where the cpd-cli command-line interface is installed and run the export command:
cpd-cli export-import export create -c <component_name> --namespace <namespace> <export_name> -f <export_yaml_file> --log-level=debug --verbose --profile=default

Replace the following values:

  • Replace component_name with the value policy-api.
  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace export_name with the name of the export task.
  • Replace export_yaml_file with the path to the YAML file with the export specification.

Example:

cpd-cli export-import export create -c policy-api --namespace wkc export1 -f /root/export.yaml --log-level=debug --verbose --profile=default

An exported .json file is created at ../data/export.json of the export package.

  1. Wait for export to complete. You can check the export status by running the following command:
cpd-cli export-import export status --namespace <namespace> <export_name> --profile=default

Replace the following values:

  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace export_name with the name of the export task.

Example of replaced values:

cpd-cli export-import export status --namespace wkc export1 --profile=default

Tip: A status of SUCCEEDED indicates only the completion of the export job. If no data was exported despite the success message, check the log files. The export might have failed, for example, due to permission errors.

  1. Download the export result:
cpd-cli export-import export download --namespace <namespace> <export_name> --profile=default

Replace the following values:

  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace export_name with the name of the export task.

Example of replaced values:

cpd-cli export-import export download --namespace wkc export1 --profile=default

The name of the export package has this format: cpd-exports-<export_name>-<timestamp>-data.tar

  1. (Optional) At any time, you can download the log file for a specific export job by using the following command:
cpd-cli export-import export logs --namespace <namespace> <export_name> --profile=default

Replace the following values:

  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace export_name with the name of the export task.

Example of replaced values:

cpd-cli export-import export logs --namespace wkc export1 --profile=default

Import

To import data protection rules to the IBM Knowledge Catalog service, complete the following steps. If you want to import from an export created on a different cluster, log in to the remote cluster before you follow the steps:

  1. Prepare a YAML file with the import specification. Create an import.yaml file with the following parameters:
  • parameter: IMPORT_HOST
    value: The hostname of the cluster from which you want to import data.
  • parameter: EXTERNAL_IMPORT
    value: true or false. This parameter is required. To import between two different systems, set this parameter to true so that the names in rules are converted to IDs in the target system at import.
  • parameter: IMPORT_COMMIT
    value: true or false. This parameter is required. To enable this IMPORT_COMMIT parameter, you must set the EXTERNAL_IMPORT parameter to true. For external imports, set this parameter to true to commit rules at import. If one rule fails to commit, the whole import fails. To get a report of rules that succeed in matching names to IDs at import without committing the rules, set this parameter to false.
  • parameter: AUTH_STRING
    value: The access token for the target system. You can specify either a bearer or ZenApiKey token. For more information about generating a ZenApiKey token, see Generating an API authorization token. For more information about generating a bearer token, see Generating a bearer token.

Make sure the key and indentation are as in the following example:

policy-api-aux:
    IMPORT_HOST: <hostname>
    EXTERNAL_IMPORT: true
    IMPORT_COMMIT: true
    AUTH_STRING: Bearer <token> | ZenApiKey <token>

If the import.yaml file is not provided, the following default values are used during import:

  • IMPORT_HOST: <hostname> - current host
  • EXTERNAL_IMPORT: true
  • IMPORT_COMMIT: true
  • AUTH_STRING: - The token is obtained by the cpd-cli for the user defined in profiles specified by the required --profile parameter.
  1. Go to the directory where the cpd-cli command-line interface is installed. Upload the downloaded .tar file from the export package by running the following command:
cpd-cli export-import export upload --namespace <namespace> -f <file_data.tar> --profile=default

Replace the following values:

  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace file_data.tar with the name of your export package.

Example of replaced values:

cpd-cli export-import export upload --namespace wkc -f cpd-exports-export1-20231128142513-data.tar --profile=default
  1. (Optional) Run the following command to verify the upload:
cpd-cli export-import export list --namespace <namespace> --log-level=debug --verbose --profile=default

Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.

Example of replaced values:

cpd-cli export-import export list --namespace wkc --log-level=debug --verbose --profile=default
  1. Run the following command to import the data to the IBM Knowledge Catalog service:
cpd-cli export-import import create --from-export <export_name> --namespace <namespace> <import_name> --log-level=debug --verbose -f <import_yaml_file> --profile=default

Replace the following values:

  • Replace export_name with the name of the export task.
  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace import_name with the name of the import task.
  • Replace import_yaml_file with the path to the YAML file with the import specification.

Example of replaced values:

cpd-cli export-import import create --from-export export1 --namespace wkc import1 --log-level=debug --verbose -f /root/import.yaml --profile=default
  1. Wait for import to complete. You can check the import status by running the following command:
cpd-cli export-import import status --namespace <namespace> <import_name> --profile=default

Replace the following values:

  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace import_name with the name of the import task.

Example of replaced values:

cpd-cli export-import import status --namespace wkc import1 --profile=default

Tip: A status of SUCCEEDED indicates only the completion of the import job. If no data was imported despite the success message, check the log files. The import might have failed, for example, due to permission errors.

  1. (Optional) Download a log file by running the following commands:
cpd-cli export-import import logs --namespace <namespace> <import name> --profile=default

Replace the following values:

  • Replace namespace with the namespace where the IBM Knowledge Catalog service is deployed.
  • Replace import_name with the name of the import task.

Example of replaced values:

cpd-cli export-import import logs --namespace wkc import1 --profile=default

Learn more

Parent topic: Administering IBM Knowledge Catalog