Db2 errors for reports and risk services

Symptoms

In rare cases, there are Db2 errors for services such as the reports and risk services. These may prevent report execution or risk event generation. When this occurs, these errors are seen in the logs for the related service:

SQLCODE=-1803, SQLSTATE=57056, SQLERRMC=NULLID.SYSSN200 0X5359534C564C3031, DRIVER=4.26.14
SQLCODE=-901, SQLSTATE=58004, SQLERRMC=Plan/Environment mismatch!, DRIVER=4.26.14

Resolving the problem

Use a bash shell to run these commands. If your are using Mac OSX (which defaults to zsh), access bash by running the bash command.

  1. Use oc to log into the cluster and then run these commands at the command line:
    echo '' > .previousGuardiumInsightsReplicas
        X=`oc get deployments,statefulsets -lproject=insights -oname`
        for i in $X; do 
            Y=`oc get $i -ogo-template='{{ .spec.replicas }}'`; 
            I=`echo $i | sed -e 's#[-\./]#_#g'`; 
            echo $I=$Y >> .previousGuardiumInsightsReplicas;
        done
    This will save the existing scaling factors for the existing stateful sets and pods to a temporary file.
  2. Inspect the .previousGuardiumInsightsReplicas file to see if it contains a list of stateful sets and deployments in the Guardium® Data Security Center environment:
    head .previousGuardiumInsightsReplicas

    The output of the above command should be similar to:

    deployment_apps_gisysqa_analytics_events=3
        deployment_apps_gisysqa_apigateway=3
        deployment_apps_gisysqa_audit=3
        deployment_apps_gisysqa_configuration=3
        deployment_apps_gisysqa_connections=3
        deployment_apps_gisysqa_cp_serviceability=1
        deployment_apps_gisysqa_dashboards=3
        deployment_apps_gisysqa_data_retention=1
        deployment_apps_gisysqa_datamart_processor=2
  3. Execute this command to scale the services down to zero pods:
    oc scale `oc get deployments,statefulsets -lproject=insights -oname` --replicas=0
  4. Log into the Db2 pod and run these commands:
    oc rsh c-<ns>-db2-db2u-0
    su - db2inst1
    db2stop force
    db2start
    db2rbind bludb -l /tmp/bind.log all
  5. After the rebind completes, exit the Db2 pod (exit) to take you back to the OpenShift® command line and bring the application pods back up by scaling them out to the number they were previously scaled to:
    source .previousGuardiumInsightsReplicas
    X=`oc get deployments,statefulsets -lproject=insights -oname`
    for i in $X; do 
        I=`echo $i | sed -e 's#[-\./]#_#g'`; 
        echo $i
        oc scale $i --replicas=${!I};
    done