Migration to Version 5.1.1 and later

The following instruction describes migration to IBM Manta Data Lineage Version 5.1.1 and later. Before running the MDI Lineage Migration Toolkit, you must complete all the steps in the prerequisites section.

Before completeing the following steps and running the migration, complete steps included in Prerequisites for running the MDI Migration Toolkit.

Prerequisites

  1. Download and extract command-line interface (cpd-cli). See, Installing the IBM Software Hub command-line interface (cpd-cli).

  2. Add cpd-cli directory to the PATH:

    # Ensure that cpd-cli is in the PATH
export PATH=${PATH}:<cpd-cli-location>

  3. Check the storage class of your Cloud Pak for Data cluster:

    oc get storageclass --all-namespaces

  4. Create a persistent volume claim (PVC) depending on the storage class of your Cloud Pak for Data cluster. See, Preparing to use the IBM Software Hub export and import utility.

  5. Create cpd-cli profile for Cloud Pak for Data admin user. See, Creating a profile to use the cpd-cli management commands.

  6. Check if the cpd-cli user and profile are created:

    cpd-cli config users list
cpd-cli config profiles list

  7. Initialize the export import utility. See, Initializing the export import utility.

  8. Download the MDI Lineage Migration Toolkit Patch from Fix Central. Enter your IBM ID credentials when prompted.

  9. Copy the MDI Lineage Migration Toolkit to ${MDIWORK_DIR} on the Cloud Pak for Data cluster.

    MDIWORK_DIR=/tmp/mdiwork
mkdir -p ${MDIWORK_DIR}
cd ${MDIWORK_DIR}

    • For 5.1.1, set the MDI_MIG_PATCH_NAME to:

      MDI_MIG_PATCH_NAME=mdi-lineage-migration-patch_5112.zip

    • For 5.1.2, set the MDI_MIG_PATCH_NAME to:

      MDI_MIG_PATCH_NAME=mdi-lineage-migration-patch_5144.zip

    • For 5.1.3, set the MDI_MIG_PATCH_NAME to:

      MDI_MIG_PATCH_NAME=mdi-lineage-migration-patch_51110.zip

  10. Install the MDI Lineage Migration Toolkit patch:

    # set the namespace
PROJECT_CPD_INST_OPERANDS=<namespace>
# set OCP node hostname and OCP port
OCP_URL=<https://<ocp> hostname>:<ocp port>
NAMESPACE=${PROJECT_CPD_INST_OPERANDS}
cd ${MDIWORK_DIR}
unzip ${MDI_MIG_PATCH_NAME}
chmod -R 755 mdi-lineage-migration-patch
cd mdi-lineage-migration-patch
./install-mdi-lineage-migration-config-spec.sh -u <username> -p <password> -url ${OCP_URL} -n ${NAMESPACE}

Prerequisites for air-gapped environment

Note: The following steps should only be used for the MDI Lineage Migration Toolkit patch in an air-gapped environment.

  1. Log in to the OpenShift console as the cluster admin.

  2. Prepare the authentication credentials to access the IBM production repository. Use the same auth.json file as for CASE download and image mirroring, or create an auth.json file that contains credentials to access icr.io and your local private registry. For more information about the auth.json file, see Syntax for the registry authentication file.

    An example directory path:

    ${HOME}/.airgap/auth.json

    An example of creating an auth.json file:

    {
"auths": {
"cp.icr.io":{"email":"unused","auth":"<base64 encoded id:apikey>"},
"<private registry hostname>":{"email":"unused","auth":"<base64 encoded
id:password>"}
}
}

  3. Install skopeo:

    yum install skopeo

  4. Confirm the path for the local private registry to copy the hotfix images to:

    oc describe pod <hotfix image pod> | grep -i "image:"

    An example:

    oc describe pod catalog-api | grep Image:
Image: cp.icr.io/cp/cpd/catalog-api@sha256:03c88c69b986f24d39e4556731c0d171169d2bd91b0fb22f6367fd51c9020e64

  5. Get the local private registry source details:

    oc get imageContentSourcePolicy
oc describe imageContentSourcePolicy [cloud-pak-for-data-mirror]

    An example output:

    - mirrors:
  - ${PRIVATE_REGISTRY_LOCATION}/cp/
  source: cp.icr.io/cp/cpd

    For more information about mirroring of images, see Mirroring IBM Software Hub images to a private container registry.

  6. Use the skopeo command to copy the patch images from the IBM production registry to the local private registry. Using the appropriate auth.json file, copy the patch images from the IBM production registry to the OpenShift cluster registry.

    DOCKER_IMAGE_ID=90d12c276279f0d7db9b885f55016ee30b571432ba1f24f9bab59fefd1fecdd1
skopeo copy --all --authfile "<folder path>/auth.json" \
    --dest-tls-verify=false --src-tls-verify=false \
    docker://cp.icr.io/cp/cpd/mdi-lineage-migration@sha256:${DOCKER_IMAGE_ID} \
    docker://${PRIVATE_REGISTRY_LOCATION}/cp/cpd/mdi-lineage-migration@sha256:${DOCKER_IMAGE_ID}

    • For 5.1.1, set the DOCKER_IMAGE_ID to:

      DOCKER_IMAGE_ID=6e6b800e8ca2d096e8b36767656ec2f335b7df009861e66355bb8f67b27713cf

    • For 5.1.2, set the DOCKER_IMAGE_ID to:

      DOCKER_IMAGE_ID=fe65fd1eb42de5643983ff76d862b084cd3cb752349a44554f7b0470325e8997

    • For 5.1.3, set the DOCKER_IMAGE_ID to:

      DOCKER_IMAGE_ID=f62503244e887776cb323d0c808cc8740b7e33145a4385332a900c3e4f0ed8bc

Running migration

Note:

Migration needs to be run as root or by a user with sudo access. See, Prerequisites for running the MDI Migration Toolkit.

  1. Add cpd-cli directory to PATH:

    export PATH=${PATH}:<cpd-cli-location>

  2. Get the list of profiles.

    cpd-cli config profiles list

  3. Set a profile name using the value from the list of profiles.

    CPD_PROFILE_NAME=<cpd-profile-name>

  4. Set environment variables for the following required parameters:

    NAMESPACE=<namespace>
CP4D_HOST=<cp4d hostname>
MIGRATION_USERNAME=<migration username>
MIGRATION_USER_APIKEY=<migration user apikey>
MIGRATION_INSTANCE_NAME=<migration instance name>
DSD_ADMIN_USERNAME=<dsd admin username>
LINEAGE_ADMIN_USERNAME=<lineage admin username>
LOG_LEVEL=<info|debug>
MDIWORK_DIR=/tmp/mdiwork

    • For MIGRATION_USERNAME parameter, provide the user with Manage projects permission. This user is also called as migration user.

    • For MIGRATION_USER_APIKEY parameter, provide the API Key of the migration user.

    • For DSD_ADMIN_USERNAME parameter, provide the user with Create data source definitions and Manage data source definitions permissions. This user needs to be added to the Platform assets catalog with the Admin role.

    • For LINEAGE_ADMIN_USERNAME parameter, provide the system admin username. The system admin user needs to have at least the following permissions: Administer platform, Manage catalogs, Manage users, Manage asset discovery, Access data lineage, Manage data lineage.

    • If you want the migration to use the system admin user to create and manage data source definitions, then both the DSD_ADMIN_USERNAME and LINEAGE_ADMIN_USERNAME parameters can use the same system admin username.

    • If you want to run the migration in debug mode, set LOG_LEVEL=debug. Otherwise, set LOG_LEVEL=info.

  5. Create migration parameters YAML file:

    MIGRATION_USER_BEARER_TOKEN=$(curl -i -s -k -X POST "https://${CP4D_HOST}/icp4d-api/v1/authorize" -H "Accept: application/json" -H "Content-Type: application/json" -d "{\"username\":\"${MIGRATION_USERNAME}\",\"api_key\":\"${MIGRATION_USER_APIKEY}\"}" | grep '"token":' | grep -o '"token":\S*"' | awk -F , '{print$1}' | awk -F : '{print$2}' | tr -d '"')
cd ${MDIWORK_DIR}
cat <<EOF > mdi-lineage-mig-params.yaml
mdi-lineage-migration-aux:
 MIGRATION_INSTANCE_NAME: ${MIGRATION_INSTANCE_NAME}
 MIGRATION_USER_BEARER_TOKEN: ${MIGRATION_USER_BEARER_TOKEN}
 DSD_ADMIN_USERNAME: ${DSD_ADMIN_USERNAME}
 LINEAGE_ADMIN_USERNAME: ${LINEAGE_ADMIN_USERNAME}
EOF

  6. Run the export using migration parameters YAML file:

    cd ${MDIWORK_DIR}
cpd-cli export-import export create -c mdi-lineage-migration -n ${NAMESPACE} --profile=${CPD_PROFILE_NAME} --backoff-limit=0 -f mdi-lineage-mig-params.yaml ${MIGRATION_INSTANCE_NAME} --log-level=${LOG_LEVEL}

  7. Check the export status and confirm that migration is completed:

    cpd-cli export-import export status -n ${NAMESPACE} ${MIGRATION_INSTANCE_NAME} --profile=${CPD_PROFILE_NAME}

    An example output:

    cpd-cli export-import export status -n ${NAMESPACE} ${MIGRATION_INSTANCE_NAME} --profile=${CPD_PROFILE_NAME}
Name:        	14febtest02                    
Job Name:    	cpd-ex-14febtest02             
Active:      	0                              
Succeeded:   	1                              
Failed:      	0                              
Start Time:  	Fri, 14 Feb 2025 00:18:42 -0800
Completed At:	Fri, 14 Feb 2025 00:21:20 -0800
Duration:    	2m38s

  8. Get the import list:

    cpd-cli export-import import list --profile=${CPD_PROFILE_NAME}

  9. Delete existing import if found in the above step:

    OLD_IMPORT_NAME=<name of the existing import>
cpd-cli export-import import delete ${OLD_IMPORT_NAME} -n ${NAMESPACE} --profile=${CPD_PROFILE_NAME}

  10. Run the import:

    cd ${MDIWORK_DIR}
cpd-cli export-import import create -n ${NAMESPACE} --profile=${CPD_PROFILE_NAME} -f mdi-lineage-mig-params.yaml ${MIGRATION_INSTANCE_NAME}im1 --backoff-limit 0 --log-level=${LOG_LEVEL} --from-export ${MIGRATION_INSTANCE_NAME}

  11. Check the import status and confirm that migration is completed:

    cpd-cli export-import import status -n ${NAMESPACE} ${MIGRATION_INSTANCE_NAME}im1 --profile=${CPD_PROFILE_NAME}

    An example output:

    cpd-cli export-import import status -n ${NAMESPACE} ${MIGRATION_INSTANCE_NAME}im1 --profile=${CPD_PROFILE_NAME}
Name:        	14febtest02im1                    
Job Name:    	cpd-im-14febtest02             
Active:      	0                              
Succeeded:   	1                              
Failed:      	0                              
Start Time:  	Fri, 14 Feb 2025 00:18:42 -0800
Completed At:	Fri, 14 Feb 2025 00:24:20 -0800
Duration:    	5m38s