MongoDB warning in pre-upgrade validation script

Symptoms

After running systest-preupgrade-validation Guardium® Insights, the log contains this warning: 
[WARNING] File MONGO_USER is not available.
Attempting to connect to db2...
Failed to update schema for tenant connect@src/mongo/shell/mongo.js:353:17,
 error: [IBM][CLI Driver][DB2/LINUXX8664] SQL0204N  "connect@src/mongo/shell/mongo.js:353:17.DM_FILE_INFO"
 is an undefined name.

Diagnosing the problem

Issue this command:

oc logs -f <mongo_pod>

Where <mongo_pod> is the MongoDB pod. You will see errors similar to:

{"t":{"$date":"2024-06-26T23:52:26.276+00:00"},"s":"I",  "c":"NETWORK",  "id":22942,  
 "ctx":"listener","msg":"Connection refused because there are too many open connections",
 "attr":{"connectionCount":901}}

Resolving the problem

Scale down the Guardium Insights deployment (but not the databases) and then proceed with upgrade:
  1. From within the CASE bundle, locate startorstopGuardiumInsights.sh.
  2. Run the script to stop the deployments: 
    ./startorstopGuardiumInsights.sh stop
  3. If upgrade job is still running: After a few iterations, it will be able to connect to MongoDB and allow the upgrade job to proceed.
  4. If the upgrade job is no longer running (no new pods for upgrade are being created): Scale down the insights operator and then scale back up by running these commands: 
    oc scale deployments ibm-guardium-insights-operator --replicas=0
oc scale deployments ibm-guardium-insights-operator --replicas=1
  5. Monitor the pods until a new upgrade pod comes up.
  6. When pre-upgrade is completed, run this command: 
    ./startorstopGuardiumInsights.sh
   start