Mirroring Data Cataloging images

Mirror the IBM® Spectrum Discover images to your enterprise registry.

About this task

For more information about Air gap setup for network restricted Red Hat® OpenShift® Container Platform clusters, see Offline setup for network restricted Red Hat OpenShift Container Platform clusters.
Note: This procedure is only necessary if you plan to enable the Data Cataloging IBM Storage Fusion HCI System service.

Procedure

  1. Run the following command to login to Docker registry with your Red Hat enterprise credentials:
    docker login registry.redhat.io -u <Red Hat enterprise registry username> -p <Red Hat enterprise registry password>
    Log in to the IBM Entitled Container Registry using the IBM entitlement key:
    docker login cp.icr.io -u cp -p <your entitlement key>
    Note: Ensure that your entitlement key for IBM Storage Fusion HCI System contains the correct entitlement.
    Set the following environment variables:
    
    export LOCAL_SECRET_JSON='<relative path to pull-secret.json>'
    export LOCAL_ISF_REGISTRY="<Your enterprise registry host>:<port>"
    export LOCAL_ISF_REPOSITORY="<Your image path>"
    IFS='/' read -r NAMESPACE PREFIX <<< "$LOCAL_ISF_REPOSITORY"
    if [[ "$PREFIX" != "" ]]; then export TARGET_PATH="$LOCAL_ISF_REGISTRY/$NAMESPACE/$PREFIX"; export REPO_PREFIX=$(echo "$PREFIX"| sed -r 's/\//-/g')-; export NAMESPACE="$NAMESPACE"; else export TARGET_PATH="$LOCAL_ISF_REGISTRY/$NAMESPACE"; export REPO_PREFIX=""; fi
    #verify both variables set correctly
    echo "$TARGET_PATH"
    echo "$NAMESPACE"
    echo "$REPO_PREFIX" 
    Note: Port is a non-mandatory value when setting the LOCAL_ISF_REGISTRY variable. You can ignore this if your enterprise registry is accessible and has a secure connection.
    Sample value for without port:
    export LOCAL_ISF_REGISTRY="registryhost.com"
    See the following sample values:
    
    export LOCAL_SECRET_JSON='/home/mirror/pull-secret.json'
    export LOCAL_ISF_REGISTRY="registryhost.com:443"
    export LOCAL_ISF_REPOSITORY="fusion-mirror"

    LOCAL_ISF_REGISTRY is your entitlement registry.

    LOCAL_ISF_REPOSITORY is the image path in which you want to mirror the images. You can choose your own repository paths. For example, hci-2.6.1/isf or hci-2.6.1.

  2. Run the command to login to the Docker registry with your enterprise registry credentials.
    docker login $LOCAL_ISF_REGISTRY -u <your enterprise registry username> -p <your enterprise registry password>
    
  3. If you have docker, run the following opm index prune command to create a new index image that contains amq-streams package:
    opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.10 -p amq-streams -t "$TARGET_PATH"/data-cataloging-redhat-operator-index:v4.10 -c docker
    Note: If you have podman instead of docker, use the following command
    opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.10 -p amq-streams -t "$TARGET_PATH"/data-cataloging-redhat-operator-index:v4.10
  4. Run the command to push the created image to enterprise registry:
    docker push "$TARGET_PATH"/data-cataloging-redhat-operator-index:v4.10
    Note: If you have podman instead of docker, use the following command
    podman push "$TARGET_PATH"/data-cataloging-redhat-operator-index:v4.10
  5. Run the command to create catalog mirror image:
    nohup oc adm catalog mirror -a "$LOCAL_SECRET_JSON" "$TARGET_PATH"/data-cataloging-redhat-operator-index:v4.10 "$TARGET_PATH" --index-filter-by-os='linux/amd64' &
    You can monitor the status in the generated nohup.out.
    wrote mirroring manifests to manifests-<index_image_name>-<random_number>
    
  6. After successfully mirroring the content to your registry, check the manifests directory that is created in your current directory. Ensure that the directory name has the following form and the generated folder should contain the following three files:
    manifests-<index_image_name>-<random_number>
    catalogSource.yaml  imageContentSourcePolicy.yaml  mapping.txt
    For example see the following folder and files generated in your current directory:
    ls manifests-redhat-operator-index-123456789
    catalogSource.yaml  imageContentSourcePolicy.yaml  mapping.txt
  7. Attention: You can skip this step, if you have already added ImageContentSourcePolicy.yaml for amq-streams during IBM Backup & Restore.
    Run the following oc command to add generated ImageContentSourcePolicy.yaml on the cluster for required Red Hat operator packages.
    Note:
    • Ensure that you complete the IBM Spectrum Protect Plus service installation, if you have planned to. In the case of an upgrade, it should first complete the upgrade. Later on, apply ImageContentSourcePolicy for the IBM Data Cataloging service install.
    • Ensure that the generated ImageContentSourcePolicy has a valid name as per the OpenShift resource naming convention. The names of resources should be up to a maximum length of 253 characters and consist of lower case alphanumeric characters, '-', and '.'.
    oc create -f ImageContentSourcePolicy.yaml
  8. From the mirroring host, run the following commands to copy images to be mirrored for Data Cataloging:
    Note: Make sure you are logged into the source and destination repositories via docker login command.
    skopeo copy --all docker://icr.io/cpopen/db2u-operator@sha256:b8a70a044e5c0217f43e945231bd5c18b06f35d29e1602083a9a84de175422af docker://$TARGET_PATH/db2u-operator@sha256:b8a70a044e5c0217f43e945231bd5c18b06f35d29e1602083a9a84de175422af
    skopeo copy --all docker://icr.io/cpopen/ibm-operator-catalog@sha256:056db327f51ccc094e72f56d24ed0fc0c333369c33e80624f2bdccf7ca813435 docker://$TARGET_PATH/ibm-operator-catalog@sha256:056db327f51ccc094e72f56d24ed0fc0c333369c33e80624f2bdccf7ca813435
    skopeo copy --all docker://icr.io/cpopen/ibm-db2uoperator-bundle@sha256:b79ee2e073753a58eaeb1f12794b14afc52c508f6f037407501096cc4a953093 docker://$TARGET_PATH/ibm-db2uoperator-bundle@sha256:b79ee2e073753a58eaeb1f12794b14afc52c508f6f037407501096cc4a953093
    skopeo copy --all docker://icr.io/db2u/db2u.graph@sha256:324b61b8ec86750cece9947eb2440a4fcd6c824f48302325fc75bb24d814e642 docker://$TARGET_PATH/db2u/db2u.graph@sha256:324b61b8ec86750cece9947eb2440a4fcd6c824f48302325fc75bb24d814e642
    skopeo copy --all docker://icr.io/db2u/db2u.auxiliary.auth@sha256:b7ae8eb8ad8e47010872a395957d76a16eead69adb0c9859dffbb05b4fb8d5d8 docker://$TARGET_PATH/db2u/db2u.auxiliary.auth@sha256:b7ae8eb8ad8e47010872a395957d76a16eead69adb0c9859dffbb05b4fb8d5d8
    skopeo copy --all docker://icr.io/db2u/db2u.rest@sha256:62c219aba9cb007455ec970d89ef1a831428fb97435ddd50ff3c0eabd132170a docker://$TARGET_PATH/db2u/db2u.rest@sha256:62c219aba9cb007455ec970d89ef1a831428fb97435ddd50ff3c0eabd132170a
    skopeo copy --all docker://icr.io/db2u/db2u.tools@sha256:e9f4457747f6696f1c1004f71cda4f5fbe1e8ebce33c36f93b7cf61009dc5191 docker://$TARGET_PATH/db2u/db2u.tools@sha256:e9f4457747f6696f1c1004f71cda4f5fbe1e8ebce33c36f93b7cf61009dc5191
    skopeo copy --all docker://icr.io/db2u/db2u.instdb@sha256:e5c116a3d5790a39704ed879db7570d257ff69167f7b3b0b7a263b3b90b659ec docker://$TARGET_PATH/db2u/db2u.instdb@sha256:e5c116a3d5790a39704ed879db7570d257ff69167f7b3b0b7a263b3b90b659ec
    skopeo copy --all docker://icr.io/db2u/db2u@sha256:1eb9acf2c13f331c71fa54d2ed9407672541eed9b099cd64ea6c36d71f4a24da docker://$TARGET_PATH/db2u/db2u@sha256:1eb9acf2c13f331c71fa54d2ed9407672541eed9b099cd64ea6c36d71f4a24da
    skopeo copy --all docker://icr.io/db2u/etcd@sha256:341acb4fd18e24221a1a13af87c852c483184616da0742a2a0ad26c8bf180d1e docker://$TARGET_PATH/db2u/etcd@sha256:341acb4fd18e24221a1a13af87c852c483184616da0742a2a0ad26c8bf180d1e
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/mo-ubi-init@sha256:c0e5dcc74add9071e15e3f4e2b6a90a9a2856be5a3c2c45b01356ca854df8ccd docker://$TARGET_PATH/ibm-spectrum-discover/mo-ubi-init@sha256:c0e5dcc74add9071e15e3f4e2b6a90a9a2856be5a3c2c45b01356ca854df8ccd
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/metaocean-api@sha256:f39f93c66ff215e88957fe9facb9bd214a828b453fce7dc05965a2ea7356a4b9 docker://$TARGET_PATH/ibm-spectrum-discover/metaocean-api@sha256:f39f93c66ff215e88957fe9facb9bd214a828b453fce7dc05965a2ea7356a4b9
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/auth@sha256:86b4c9e8cab881ca6d23d391a2413bc203a9045e6023ea735542a13fff87e576 docker://$TARGET_PATH/ibm-spectrum-discover/auth@sha256:86b4c9e8cab881ca6d23d391a2413bc203a9045e6023ea735542a13fff87e576
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/backup-restore@sha256:00b78a0afe874d0f6e5d9d5857445ebcb93e71c77be6fe749835d06b3245a9ed docker://$TARGET_PATH/ibm-spectrum-discover/backup-restore@sha256:00b78a0afe874d0f6e5d9d5857445ebcb93e71c77be6fe749835d06b3245a9ed
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/connmgr@sha256:2a4ea132ab2b91e8d059d862c4a122df4d7f143d3a9b59766bd4cc91dfd934c4 docker://$TARGET_PATH/ibm-spectrum-discover/connmgr@sha256:2a4ea132ab2b91e8d059d862c4a122df4d7f143d3a9b59766bd4cc91dfd934c4
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/connmgr-scheduler@sha256:d786380884eab14767c7d7c4a0e5439d5d345c820bc8133fe1793cb6726a2645 docker://$TARGET_PATH/ibm-spectrum-discover/connmgr-scheduler@sha256:d786380884eab14767c7d7c4a0e5439d5d345c820bc8133fe1793cb6726a2645
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/moconsumer@sha256:c7fe577fb1a6a6cc8a196906580af4c4142db1eb7188c4bb333914a384370447 docker://$TARGET_PATH/ibm-spectrum-discover/moconsumer@sha256:c7fe577fb1a6a6cc8a196906580af4c4142db1eb7188c4bb333914a384370447
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/mo-agent-extract@sha256:ea334594ecc32f98cdc2f970566961e16a3633e6cce03f2acba465a33591fbe3 docker://$TARGET_PATH/ibm-spectrum-discover/mo-agent-extract@sha256:ea334594ecc32f98cdc2f970566961e16a3633e6cce03f2acba465a33591fbe3
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/db-schema@sha256:ce414d6530ec3f16a3778c4016d2e84681bed34f7d387d29589a9183c55fcb43 docker://$TARGET_PATH/ibm-spectrum-discover/db-schema@sha256:ce414d6530ec3f16a3778c4016d2e84681bed34f7d387d29589a9183c55fcb43
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/db2whrest@sha256:c8e0852809d33413d2a52dedc35966c01a0dc1c71e4fd88306a0b7766811dd18 docker://$TARGET_PATH/ibm-spectrum-discover/db2whrest@sha256:c8e0852809d33413d2a52dedc35966c01a0dc1c71e4fd88306a0b7766811dd18
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/import-tags-app@sha256:5a59884008e59a6dfd6e02c6abe99e144eb6564daf5818708dbe7d356093668e docker://$TARGET_PATH/ibm-spectrum-discover/import-tags-app@sha256:5a59884008e59a6dfd6e02c6abe99e144eb6564daf5818708dbe7d356093668e
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/keystone@sha256:13aeaa84391227d5a1eff6bb416f743a0311776ffab53e71863adc627dc8f37c docker://$TARGET_PATH/ibm-spectrum-discover/keystone@sha256:13aeaa84391227d5a1eff6bb416f743a0311776ffab53e71863adc627dc8f37c
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/policyengine@sha256:6b16017d7949ba59896728edddaf9a109288633a9fd616212995733c274ce655 docker://$TARGET_PATH/ibm-spectrum-discover/policyengine@sha256:6b16017d7949ba59896728edddaf9a109288633a9fd616212995733c274ce655
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/moproducer@sha256:bd2a2ba826050597f9311b89af85380fd4324c24f7e6080a484af70b2c4f2666 docker://$TARGET_PATH/ibm-spectrum-discover/moproducer@sha256:bd2a2ba826050597f9311b89af85380fd4324c24f7e6080a484af70b2c4f2666
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/reports@sha256:ca1cdbcedc9655d9c741d86ad9071f26d33595d7955d1430b9a3825477e7c3b2 docker://$TARGET_PATH/ibm-spectrum-discover/reports@sha256:ca1cdbcedc9655d9c741d86ad9071f26d33595d7955d1430b9a3825477e7c3b2
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/sdmonitor@sha256:d3e9330ac505bd634991d8f0c1b0a68636cb72e8c004b3ed0a623d09babd4193 docker://$TARGET_PATH/ibm-spectrum-discover/sdmonitor@sha256:d3e9330ac505bd634991d8f0c1b0a68636cb72e8c004b3ed0a623d09babd4193
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/tikaserver@sha256:e098b1dc4a4c42f406238bc1bb63843c5e162556a92f270150e80e9de1fe04bd docker://$TARGET_PATH/ibm-spectrum-discover/tikaserver@sha256:e098b1dc4a4c42f406238bc1bb63843c5e162556a92f270150e80e9de1fe04bd
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/uifrontend@sha256:b9efa7e0932113d39b8576d51150af0ee2fdeaed47ea1ac9faa66a388f803891 docker://$TARGET_PATH/ibm-spectrum-discover/uifrontend@sha256:b9efa7e0932113d39b8576d51150af0ee2fdeaed47ea1ac9faa66a388f803891
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/uibackend@sha256:47c7d6b5e884a60a95cc3dc4511e89b689ebb0fad03f2da7b56fae3f702dfa63 docker://$TARGET_PATH/ibm-spectrum-discover/uibackend@sha256:47c7d6b5e884a60a95cc3dc4511e89b689ebb0fad03f2da7b56fae3f702dfa63
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/wkc-connector@sha256:460dfd8e06242278246e55ff8241ad8af5ea8f747e86b5b6f283874c18bbc75a docker://$TARGET_PATH/ibm-spectrum-discover/wkc-connector@sha256:460dfd8e06242278246e55ff8241ad8af5ea8f747e86b5b6f283874c18bbc75a
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/scaleafmdatamover@sha256:d485870db5d654b1eb403420fc772ab8a2e14ac8ed71607123df5e5a35fa274b docker://$TARGET_PATH/ibm-spectrum-discover/scaleafmdatamover@sha256:d485870db5d654b1eb403420fc772ab8a2e14ac8ed71607123df5e5a35fa274b
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/scaleilmdatamover@sha256:cbf362fb09c977591dd92c7671a8fa16467d8026cdf7d63695fa2ebb7cb005b5 docker://$TARGET_PATH/ibm-spectrum-discover/scaleilmdatamover@sha256:cbf362fb09c977591dd92c7671a8fa16467d8026cdf7d63695fa2ebb7cb005b5
    skopeo copy --all docker://cp.icr.io/cp/ibm-spectrum-discover/isd-proxy@sha256:b27811ad3549001f9d2caa05526b42fd2f9aa9ef62d22d1f9a627331a99a916e docker://$TARGET_PATH/ibm-spectrum-discover/isd-proxy@sha256:b27811ad3549001f9d2caa05526b42fd2f9aa9ef62d22d1f9a627331a99a916e
    skopeo copy --all docker://icr.io/cpopen/ibm-spectrum-discover-operator@sha256:08b1530d286dae67238288e6b179874bec1653643b1fa674a4be0519b63e2af4 docker://$TARGET_PATH/ibm-spectrum-discover-operator@sha256:08b1530d286dae67238288e6b179874bec1653643b1fa674a4be0519b63e2af4
    skopeo copy --all docker://icr.io/cpopen/ibm-spectrum-discover-operator-bundle@sha256:78e7c3739c6c7d027df82d842acb1b12fb190155d2f96359138fa2c405be221b docker://$TARGET_PATH/ibm-spectrum-discover-operator-bundle@sha256:78e7c3739c6c7d027df82d842acb1b12fb190155d2f96359138fa2c405be221b
    skopeo copy --all docker://icr.io/cpopen/ibm-spectrum-discover-operator-catalog@sha256:d57053f13aa687fdd30e6858a31645ff8e80c3f03e02cf1ad2d42432b465e80f docker://$TARGET_PATH/ibm-spectrum-discover-operator-catalog@sha256:d57053f13aa687fdd30e6858a31645ff8e80c3f03e02cf1ad2d42432b465e80f
    

    Ensure all commands are successful.

  9. After all commands are successfully executed, create the following CatalogSource:
    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: redhat-operators-data-cataloging
      namespace: openshift-marketplace
    spec:
      displayName: Red Hat Operators for Data Cataloging
      image: "$PRUNED_RED_HAT_OPERATORS_IMAGE"
      sourceType: grpc
    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: ibm-operator-catalog
      namespace: openshift-marketplace
    spec:
      displayName: IBM Operator Catalog
      image: "$TARGET_PATH/ibm-operator-catalog@sha256:056db327f51ccc094e72f56d24ed0fc0c333369c33e80624f2bdccf7ca813435"
      sourceType: grpc
  10. Add ImageContentSourcePolicy after skopeo commands are executed successfully on the cluster.
    See the following ImageContentSourcePolicy:
    apiVersion: operator.openshift.io/v1alpha1
    kind: ImageContentSourcePolicy
    metadata:
      labels:
        operators.openshift.org/catalog: "true"
      name: isd-mirror
    spec:
      repositoryDigestMirrors:
      - mirrors:
        - $TARGET_PATH
        source: icr.io/cpopen
      - mirrors:
        - $TARGET_PATH
        source: registry.redhat.io/redhat
      - mirrors:
        - $TARGET_PATH/amq
        - $TARGET_PATH
        source: registry.redhat.io/amq7
      - mirrors:
        - $TARGET_PATH/ibm-spectrum-discover
        source: cp.icr.io/cp/ibm-spectrum-discover
      - mirrors:
        - $TARGET_PATH/db2u
        source: icr.io/db2u