Guardium Data Security Center upgrade becomes
stuck with MongoDBCommunity
error
Symptoms
Guardium Data Security Center upgrade becomes stuck with this error:MongoDBCommunity does not have phase==Running
In this state,
oc get mongodbcommunity
returns a MongoDB instance in pending
state, and oc get pods | grep mongo
shows
one or more pods with a crashloop
error. When issuing oc logs
for
this pod, there will be errors similar to
this:
{"error":"UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document
(ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version:
{ _id: \"featureCompatibilityVersion\", version: \"4.2\" }. See
https://docs.mongodb.com/master/release-notes/4.4-compatibility/#feature-compatibility.
:: caused by :: Invalid value for featureCompatibilityVersiondocument in admin.system.version,
found 4.2, expected '4.4' or '4.9' or '5.0.
Resolving the problem
- Scale down the Guardium Data Security Center
operator:
oc scale deployment ibm-guardium-insights-operator --replicas=0
- Update the
mongodbcommunity
custom resource (CR) (oc edit mongodbcommunity <name>
). Look for and changespec.featureCompatibilityVersion
to4.4
. Look for and changespec.version
to4.4.26
. Also look for one of these lines:image: <url>/mongo:5.0.24
or
In the line, replace everything afterimage: <url>/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
mongo
with this:
For example, the end result should look like this:@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
image: <url>/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
Optionally, you can run this command in your Guardium Data Security Center namespace. This will require jq binary at version 1.6:
oc get $(oc get mongodbcommunity --no-headers -oname) -ojson | jq .spec.featureCompatibilityVersion='"4.4"' | jq .spec.version='"4.4.26"' | sed -e 's#/mongo:5.0.24#/mongo:4.4.26#g' -e 's#/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e#/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75#g' | oc apply -f -
- In the MongoDB STS, update the MongoDB container image to
sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
by issuing these steps:- Issue this command:
Sample output is:oc get sts | grep mongodb
gi-mongodb 3/3 18h
- Issue this command:
This will open the file in an editor. In the editor, look for theoc edit sts gi-mongodb
image:
field. The field's value will bemongo:5.0.24
ormongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
, but not both. - If the value is
mongo:5.0.24
, change it tomongo:4.4.26
. - If the value is
mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
, change it tomongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
. - Save the file and exit.
- Issue this command:
- The pods should now appear as
running
andmongodbcommunity
Version 4.4.26 should now be successfully running on the system. When you see MongoDB 4.4.26 is running, proceed to the 5.0.24 upgrade.Note: If this step takes a long time, manually restart the MongoDB operator and pods. - Update the
mongodbcommunity
custom resource (CR) (oc edit mongodbcommunity <name>
). Look for and changespec.featureCompatibilityVersion
to5.0
. Look for and changespec.version
to5.0.24
. Also look for one of these lines:image: <url>/mongo:4.4.26
or
In the line, replace everything afterimage: <url>/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
mongo
with this:
For example, the end result should look like this:@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
image: <url>/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
Optionally, you can run this command in your Guardium Data Security Center namespace. This will require jq binary at version 1.6:
oc get $(oc get mongodbcommunity --no-headers -oname) -ojson | jq .spec.featureCompatibilityVersion='"5.0"' | jq .spec.version='"5.0.24"' | sed -e 's#/mongo:4.4.26#/mongo:5.0.24#g' -e 's#/mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75#/mongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e#g' | oc apply -f -
- In the MongoDB STS, update the MongoDB container image to
sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
.- Issue this command:
Sample output is:oc get sts | grep mongodb
gi-mongodb 3/3 18h
- Issue this command:
This will open the file in an editor. In the editor, look for theoc edit sts gi-mongodb
image:
field. The field's value will bemongo:4.4.26
ormongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
, but not both. - If the value is
mongo:4.4.26
, change it tomongo:5.0.24
. - If the value is
mongo@sha256:ffb01941382111a8c6a4454a16dd7e4377c2c2f2d4ae7c42d99cabe2744b2d75
, change it tomongo@sha256:005368eded60ec7519a6f8c0f1cba12a2ce49d52b047f556c7457376ff247f2e
. - Save the file and exit.
- Issue this command:
- The new pods should appear as
running
andmongodbcommunity
Version 5.0.24 should now be successfully running on the system.Note: If this step takes a long time, manually restart the MongoDB operator and pods. - Scale up the Guardium Data Security Center
operator:
oc scale deployment ibm-guardium-insights-operator --replicas=1