Migrating catalog assets (Watson Knowledge Catalog)

Prerequisites

Required roles

To complete this task, you must have one of these roles:

• Cluster administrator

• Instance administrator

Limitations

• Catalog access control and settings cannot be exported or imported.

• Asset membership and asset ownership cannot be imported. Migrated assets will be owned by the administrator who performs the migration. You must specify the admin_username in the import yaml file to the name of the user that acts as instance administrator, otherwise admin user is assigned as owner by default even if that user does not exist. Note that if you are using data protection rules, any data protection rules that include asset owner as a condition cease to function after the assets are migrated.

• Asset to governance artifact relationships are imported as broken relationships.

• Cross-container relationships and column relationships are not supported.

• Asset to glossary relationships cannot be exported or imported.

• Assets notes cannot be migrated.

• Assets reviews and ratings cannot be migrated.

• Term and classification assignments cannot be imported.

Before you begin Before you can use the export import utility, complete the following tasks:

1. Ensure that you created a cpd-cli profile on your workstation. For more information, see Creating a profile to use the cpd-cli management commands.

2. Ensure that the required persistent volume claim (PVC) exists in the operands project for the instance oc get pvc export-import-pvc --namespace=${PROJECT_CPD_INST_OPERANDS}. If the PVC doesn't exist, see Preparing to use the Cloud Pak for Data export and import utility.

3. Initialize the export import utility on your workstation. For more information, see Initializing the export import utility.

4. Ensure that the catalog-api-aux auxiliary module is available by running the following command:

   cpd-cli export-import list aux-modules \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --profile=${CPD_PROFILE_NAME} \
   --arch=${CPU_ARCH}
   

Supported asset types

You can export and import the following catalog assets:

• connection
• data_asset
• BI reports
• Logical data models (LDMs)
• Physical data models (PDMs)
• Custom asset types

Selecting the scope of the export

If you want to export all assets from all catalogs, projects, and spaces, go to Exporting catalog assets.

If you want to select which assets to export, use the following steps:

1. Define the name of the YAML file:

   export OVERRIDE_YAML_FILE=<fully-qualified-file-name>
   

2. Create an export.yaml file with the following structure:

   catalog-api-aux:
   # exportspec specifies which assets to export
   exportspec: 'EXPORT_JSON_STRING'
   

3. Specify which assets to export by replacing the EXPORT_JSON_STRING variable with a JSON string. If you want to export all assets from a certain workspace, your string would resemble this example:

   catalog-api-aux:
   exportspec: '{"catalog": {    "container_specs": [{   "guids": ["08f248d3-0c5f-4193-a16d-e3c0cc7f7d94"],     "all_assets": true}  ]} }'
   

   Attention: The JSON string must be placed in one line and wrapped with single quotation mark characters. For clarity, the following examples show JSON strings on separate lines.

   If you want to export certain assets, use the code in following examples:

   • Export all assets from the list of specified catalogs, projects, or spaces:

     '{
       "project": { <<---- this is the type of asset container: catalog, project, or space
         "container_specs": [
           {
             "guids": [
               "00cf5b102-17b3-4638-95df-309a2a443137",  <<---- these are the ids of asset containers
               "e14a187d-4fc8-4a5d-aee1-e2ad4ec60b29"
             ],
             "all_assets": true
           }
         ]
       },
       "catalog": { <<---- this is the type of asset container: catalog, project, or space
         "container_specs": [
           {
             "guids": [
               "ab3d18a4-015f-4bb1-ae13-4ee9faf0382a"  <<---- these are the ids of asset containers
             ],
             "all_assets": true
           }
         ]
       }
     }'
     

   • Export specific assets from the list of specified catalogs, projects, or spaces:

     '{
       "space": { <<---- this is the type of asset container: catalog, project, or space
         "container_specs": [
           {
             "guids": [
               "4b014869-fc0e-4919-bc93-3fa8c40bcd37"  <<---- this is the id of asset container
             ],
             "asset_specs": {
               "asset_ids": [
                 "e14a187d-4fc8-4a5d-aee1-e2ad4ec60b29",  <<---- these are the ids of assets
                 "e2395865-5503-49da-b708-cfbf2fb1b646"
               ]
             }
           }
         ]
       }
     }'
     

   • Export assets of specific types in from the list of specified catalogs, projects, or spaces:

     '{
       "space": { <<---- this is the type of asset container: catalog, project, or space
         "container_specs": [
           {
             "guids": [
               "4b014869-fc0e-4919-bc93-3fa8c40bcd37",  <<---- these are the ids of asset containers
               "866613f8-4102-4cd2-be6e-ee46ea6c468b"
             ],
             "asset_specs": {
               "asset_types": [
                 "data_asset",  <<---- these are the types of assets
                 "notebook"
               ]
             }
           }
         ]
       }
     }'
     

Exporting catalog assets

To export catalog assets from the Watson Knowledge Catalog service, complete the following tasks:

1. Define the name of the export task:

   export EXPORT_NAME=<name>
   

2. Go to the directory where the cpd-cli command-line interface is installed and run the appropriate export command for your scope:

   • If you want to export all assets from all catalogs, projects, and spaces, create an export.yaml file with the following structure:

     ./cpd-cli export-import export create ${EXPORT_NAME} -n ${PROJECT_CPD_INST_OPERANDS} \
       --component catalog-api
     

   • If you want to select which assets to export, create an export.yaml file with the following structure:

     ./cpd-cli export-import export create ${EXPORT_NAME} -n ${PROJECT_CPD_INST_OPERANDS} \
       --component catalog-api \
       --values ${OVERRIDE_YAML_FILE}
     

3. Wait for the export to complete. You can check the export status by using this command:

   ./cpd-cli export-import export status ${EXPORT_NAME} -n ${PROJECT_CPD_INST_OPERANDS}
   

4. Download the export result:

   ./cpd-cli export-import export download ${EXPORT_NAME} -n ${PROJECT_CPD_INST_OPERANDS}
   

   The name of the exported package has this format: cpd-exports-<export_name>-<timestamp>-data.tar

5. Set an environment variable to define the export package name:

   export WKC_ASSET_EXPORT=<export_file_name>
   

Selecting the scope of the import

If you want to import all assets from all catalogs, projects, and spaces, go to step Importing catalog assets.

If you want to select which assets to import, create a YAML file with the import specification, use the following steps:

1. Create an import.yaml file with the following structure:

   catalog-api-aux:
     # importspec specifies which assets to import
     importspec: 'IMPORT_JSON_STRING'
   

2. Replace the IMPORT_JSON_STRING variable with a JSON string that specifies the assets to import.

   Example:

   catalog-api-aux:
     importspec: '{"catalog": {    "container_specs": [{   "guids": ["08f248d3-0c5f-4193-a16d-e3c0cc7f7d94"],    "all_assets": true}  ]} }'
   

   You can add the following import specifications to your YAML file to select which assets to import:

   • containerIds: import-specific source catalogs into specific existing containers in the new cluster. The value is a JSON map of source: target.

     Example:

     '{ "containerIds": { "default": "<target-id1>", "<original-id>": "<target-id2>" } }'
     

   • newContainerNameOverrides: import containers into new containers. You can give specific names to containers before they are created. Otherwise, new containers are given default names. The value is a JSON map of source: target.

     Example:

     { "newContainerNameOverrides": { "<source-id1>": "<target-name1>", "<source-id2>": "<target-name2>" } }
     

   • members: set the default membership of the imported assets when a catalog is imported. The catalog is automatically updated with its own membership settings that achieve the desired asset membership settings. Previous membership information is removed in favor of setting provided groups as members.

     Example:

     catalog-api-aux:
      admin_username: test-user-dd
      importspec:  importspec: '{ "containerIds": {"default": "7345d4bc-b377-41e1-98b2-1e79767f8d97"}, "members":    { "default": [{"access_group_id": "10010", "roles": ["VIEWER"]}, {"access_group_id": "20010", "roles":   ["EDITOR"]}, {"access_group_id": "30010", "roles": ["OWNER"]}]} }'
     

   • duplicate_action: specify which action to take during asset creation. Choose one of 4 values: "UPDATE", "REJECT", "REPLACE", or "IGNORE".

     Attention:

     duplicate_action is not supported for the connection asset type.

     Example:

     { "duplicate_action": "REPLACE" }
     

Importing catalog assets

To import catalog assets to the Watson Knowledge Catalog service, complete the following tasks. If you want to import from an export created on a different cluster, copy exported archive to the current cluster, and log in to the current cluster.

1. Go to the directory where the cpd-cli command-line interface is installed and upload the exported package:

   ./cpd-cli export-import export upload -n ${PROJECT_CPD_INST_OPERANDS} \
   -f Export_archive_file_name
   

2. Run the following command to import the data to the Watson Knowledge Catalog service:

   ./cpd-cli export-import import create Import_name -n ${PROJECT_CPD_INST_OPERANDS} \
    --from-export ${EXPORT_NAME}
   

3. Wait for import to complete. You can check the import status by using this command:

   ./cpd-cli export-import import status Import_name -n ${PROJECT_CPD_INST_OPERANDS}
   

Learn more

• Migrating data between Cloud Pak for Data installations

Parent topic: Administering Watson Knowledge Catalog