Guardium Data Security Center upgrade becomes stuck with MongoDBCommunity error

Edit online

Symptoms

Guardium® Data Security Center upgrade becomes stuck with this error:
MongoDBCommunity does not have phase==Running

In this state, oc get mongodbcommunity returns a MongoDB instance in pending state, and oc get pods | grep mongo shows one or more pods with a crashloop error. When issuing oc logs for this pod, there will be errors similar to this:

{"error":"UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document 
(ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: 
{ _id: \"featureCompatibilityVersion\", version: \"4.2\" }. See 
https://docs.mongodb.com/master/release-notes/4.4-compatibility/#feature-compatibility. 
:: caused by :: Invalid value for featureCompatibilityVersiondocument in admin.system.version, 
found 4.2, expected '4.4' or '4.9' or '5.0.

Resolving the problem

  1. Scale down the Guardium Data Security Center operator:
    oc scale deployment ibm-guardium-insights-operator --replicas=0
  2. Update the mongodbcommunity custom resource (CR) (oc edit mongodbcommunity <name>). Look for and change spec.featureCompatibilityVersion to 4.4. Look for and change spec.version to 4.4.26. Also look for one of these lines:
    image: <url>/mongo:5.0.24

    or

    image: <url>/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
    In the line, replace everything after mongo with this:
    @sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
    For example, the end result should look like this:
    image: <url>/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75

    Optionally, you can run this command in your Guardium Data Security Center namespace. This will require jq binary at version 1.6:

    oc get $(oc get mongodbcommunity --no-headers -oname) -ojson | jq .spec.featureCompatibilityVersion='"4.4"' | 
jq .spec.version='"4.4.26"' | sed -e 's#/mongo:5.0.24#/mongo:4.4.26#g' 
-e 's#/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e#/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75#g' 
| oc apply -f -
  3. In the MongoDB STS, update the MongoDB container image to sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75 by issuing these steps:
    1. Issue this command:
      oc get sts | grep mongodb
      Sample output is: gi-mongodb 3/3 18h
    2. Issue this command:
      oc edit sts gi-mongodb
      This will open the file in an editor. In the editor, look for the image: field. The field's value will be mongo:5.0.24 or mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e, but not both.
    3. If the value is mongo:5.0.24, change it to mongo:4.4.26.
    4. If the value is mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e, change it to mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75.
    5. Save the file and exit.
  4. The pods should now appear as running and mongodbcommunity Version 4.4.26 should now be successfully running on the system. When you see MongoDB 4.4.26 is running, proceed to the 5.0.24 upgrade.
    Note: If this step takes a long time, manually restart the MongoDB operator and pods.
  5. Update the mongodbcommunity custom resource (CR) (oc edit mongodbcommunity <name>). Look for and change spec.featureCompatibilityVersion to 5.0. Look for and change spec.version to 5.0.24. Also look for one of these lines:
    image: <url>/mongo:4.4.26

    or

    image: <url>/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
    In the line, replace everything after mongo with this:
    @sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
    For example, the end result should look like this:
    image: <url>/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e

    Optionally, you can run this command in your Guardium Data Security Center namespace. This will require jq binary at version 1.6:

    oc get $(oc get mongodbcommunity --no-headers -oname) -ojson | jq .spec.featureCompatibilityVersion='"5.0"' | 
jq .spec.version='"5.0.24"' | sed -e 's#/mongo:4.4.26#/mongo:5.0.24#g' 
-e 's#/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75#/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e#g' 
| oc apply -f -
  6. In the MongoDB STS, update the MongoDB container image to sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e.
    1. Issue this command:
      oc get sts | grep mongodb
      Sample output is: gi-mongodb 3/3 18h
    2. Issue this command:
      oc edit sts gi-mongodb
      This will open the file in an editor. In the editor, look for the image: field. The field's value will be mongo:4.4.26 or mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75, but not both.
    3. If the value is mongo:4.4.26, change it to mongo:5.0.24.
    4. If the value is mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75, change it to mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e.
    5. Save the file and exit.
  7. The new pods should appear as running and mongodbcommunity Version 5.0.24 should now be successfully running on the system.
    Note: If this step takes a long time, manually restart the MongoDB operator and pods.
  8. Scale up the Guardium Data Security Center operator:
    oc scale deployment ibm-guardium-insights-operator --replicas=1