Missing MongoDB image results in upgrade and installation failure

Symptoms

When upgrading, patching, or installing Guardium® Data Security Center, there is a failure to pull the MongoDB image:

> oc get po | grep mongo
mongodb-kubernetes-operator-7db6fc94950-zwndn       1/1   Running                 0
gi-sample-mongodb-0                                 0/2   Init:ImagePullBackOff   0
gi-sample-mongodb-certconvert-2fnhs                 0/1   Completed               0
gi-sample-mongodb-ready-6mt16                       0/1   Error                   0
gi-sample-mongodb-ready-nkd22                       0/1   Error                   0
gi-sample-mongodb-ready-sd4z8                       1/1   Running                 0

> oc describe po gi-sample-mongodb-0
.....
 Normal  Pulling  6m45s (x4 over 8m6s)  kubelet      Pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc"
 Warning Failed   6m44s (x4 over 8m5s)  kubelet      Failed to pull image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc": rpc error: code = NotFound desc = failed to pull and unpack image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc": failed to resolve reference "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc": quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc: not found
 Warning Failed   6m44s (x4 over 8m5s)  kubelet      Error: ErrImagePull
 Warning Failed   6m18s (x6 over 8m4s)  kubelet      Error: ImagePullBackOff
 Normal  BackOff  2m58s (x20 over 8m4s) kubelet      Back-off pulling image "quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:ce2a09a3005735db270c940f7b12a1b7dd40961d7f47cd1ce9f8a70e4fa057dc"

Causes

The mongodb-kubernetes-operator-version-upgrade-post-start-hook image is no longer available in the https://quay.io/ image registry.

Environment

This occurs when upgrading to or during a fresh installation of version 3.6.0 - and it occurs when patching to version 3.6.1.

Resolving the problem

For air gap installation, update the image registry to mirror the missing image SHA for mongodb-kubernetes-operator-version-upgrade-post-start-hook by running these commands:

oc image mirror \
  icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:f60f9e37e81c5d853af11f5a4b5497443b256beb64d67f8125ebd3f3cc7b1972 \
  $PRIVATE_DOCKER_REGISTRY/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9

# Patch the mongoDB Statefull set
export gdscname=<gdsc-instance-name> # e.g. gi-sample
export namespace=<gdsc-namespace> # e.g. gi-sample

oc patch deployment mongodb-kubernetes-operator\
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/0/env/6/value",
      "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"
    }
  ]'

oc patch sts ${gdscname}-mongodb -n ${namespace} --type=json -p '[{"op": "replace", "path": "/spec/template/spec/initContainers/0/image", "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:f60f9e37e81c5d853af11f5a4b5497443b256beb64d67f8125ebd3f3cc7b1972"}]'


oc get pods | grep "${gdscname}-mongodb-[0-2]" | awk '{print $1}' | xargs oc delete pod

For online installation, update the image registry to mirror the missing image tag for mongodb-kubernetes-operator-version-upgrade-post-start-hook by running these commands:

# Patch the mongoDB Statefull set
export gdscname=<gdsc-instance-name> # e.g. gi-sample
export namespace=<gdsc-namespace> # e.g. gi-sample


oc patch deployment mongodb-kubernetes-operator\
  --type='json' \
  -p='[
    {
      "op": "replace",
      "path": "/spec/template/spec/containers/0/env/6/value",
      "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"
    }
  ]'


oc patch sts ${gdscname}-mongodb -n ${namespace} --type=json -p '[{"op": "replace", "path": "/spec/template/spec/initContainers/0/image", "value": "icr.io/cpopen/ibm-guardium-data-security-center/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9"}]' 


oc get pods | grep "${gdscname}-mongodb-[0-2]" | awk '{print $1}' | xargs oc delete pod