IBM Support

Missing MongoDB image results in Guardium Data Security Center installation failure

Troubleshooting


Problem

When installing Guardium Data Security Center, there is a failure to pull the MongoDB image.

Symptom

Failure to pull the MongoDB image errors:
  
> oc get po | grep mongo
mongodb-kubernetes-operator-7db6fc94950-zwndn       1/1   Running                 0
gi-sample-mongodb-0                                 0/2   Init:ImagePullBackOff   0
gi-sample-mongodb-certconvert-2fnhs                 0/1   Completed               0
gi-sample-mongodb-ready-6mt16                       0/1   Error                   0
gi-sample-mongodb-ready-nkd22                       0/1   Error                   0
gi-sample-mongodb-ready-sd4z8                       1/1   Running                 0

> oc describe po gi-sample-mongodb-0
.....
 Normal  Pulling  6m45s (x4 over 8m6s)  kubelet      Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc"
 Warning Failed   6m44s (x4 over 8m5s)  kubelet      Failed to pull image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc": rpc error: code = NotFound desc = failed to pull and unpack image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc": failed to resolve reference "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc": quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc: not found
 Warning Failed   6m44s (x4 over 8m5s)  kubelet      Error: ErrImagePull
 Warning Failed   6m18s (x6 over 8m4s)  kubelet      Error: ImagePullBackOff
 Normal  BackOff  2m58s (x20 over 8m4s) kubelet      Back-off pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc"

Cause

The mongodb-kubernetes-operator-version-upgrade-post-start-hook image is no longer available in the https://quay.io/ image registry.

Environment

This occurs when upgrading to 3.6.0 or patching to 3.6.1 - or during a fresh installation of versions 3.6.0 or 3.6.1.

Resolving The Problem

  • For air gap installation, update the image registry to mirror the missing image SHA for mongodb-kubernetes-operator-version-upgrade-post-start-hook by running these commands: 
    oc image mirror \
  icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:f60f9e37e81c5d853af11f5a4b5497443b256beb64d67f8125ebd3f3cc7b1972 \
  $PRIVATE_DOCKER_REGISTRY/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9

# Patch the mongoDB Statefull set
export gdscname=<gdsc-instance-name> # e.g. gi-sample
export namespace=<gdsc-namespace> # e.g. gi-sample

oc patch deployment mongodb-kubernetes-operator\
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/0/env/6/value",
      "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"
    }
  ]'

oc patch sts ${gdscname}-mongodb -n ${namespace} --type=json -p '[{"op": "replace", "path": "/spec/template/spec/initContainers/0/image", "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:f60f9e37e81c5d853af11f5a4b5497443b256beb64d67f8125ebd3f3cc7b1972"}]'


oc get pods | grep "${gdscname}-mongodb-[0-2]" | awk '{print $1}' | xargs oc delete pod
  • For online installation, update the image registry to mirror the missing image tag for mongodb-kubernetes-operator-version-upgrade-post-start-hook by running these commands: 
    # Patch the mongoDB Statefull set
export gdscname=<gdsc-instance-name> # e.g. gi-sample
export namespace=<gdsc-namespace> # e.g. gi-sample


oc patch deployment mongodb-kubernetes-operator\
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/0/env/6/value",
      "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"
    }
  ]'


oc patch sts ${gdscname}-mongodb -n ${namespace} --type=json -p '[{"op": "replace", "path": "/spec/template/spec/initContainers/0/image", "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"}]' 


oc get pods | grep "${gdscname}-mongodb-[0-2]" | awk '{print $1}' | xargs oc delete pod

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSJGT3M","label":"IBM Guardium Data Security Center"},"ARM Category":[{"code":"a8m0z000000Gp0NAAS","label":"INSTALL UPGRADE MIGRATION"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"3.6.0;3.6.1"}]

Document Information

Modified date:
13 February 2025

UID

ibm17180501

Page Feedback