IBM Support

isc-cases-activemq is Restarting or in Error Status

Troubleshooting


Problem

UI displays upstream connect error:
upstream connect error or disconnect/reset before headers. reset reason: connection failure.

Symptom

oc get pods
isc-cases-activemq in error or CrashLoopBackOff status

Cause

Possible redeployment of foundations, solutions charts, or crunchy data might cause certificate or secrets to be null. 

Diagnosing The Problem

  1. Perform 
    kubectl logs isc-cases-activemq*
    NOTE: (Activemq is full name from oc get pods)
  2. Check error messages contain secret or SSL cert
  3. Ensure to note days next to secrets. Check if isc-cases-postgres-* secrets are older than the isc-cases-db-passwords: 
    oc get secrets | grep cases
  4. kubectl logs idc-cases* might display the following as the cause: 
    Importing trusted certs /certs/postgres/postgres_cert /certs/icd_public
Importing certificate file /certs/postgres/postgres_cert into keystore
Certificate was added to keystore
cert file /certs/icd_public not present
{DATE AND TIME} org.postgresql.Driver connect
SEVERE: Connection error: 
org.postgresql.util.PSQLException: SSL error: com.ibm.jsse2.util.h: PKIX path validation failed: java.security.cert.CertPathValidatorException: The certificate issued by CN=isc-cases-postgres is not trusted; internal cause is: 
	java.security.cert.CertPathValidatorException: Certificate chaining error
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:67)
	at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:359)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:148)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)

Resolving The Problem

Regenerate cm - isc-cases-pgcluster-configmap and isc-cases-postgres-ca-cert to solve this issue.
  1. Delete currant certs: 
    oc delete cm isc-cases-pgcluster-configmap isc-cases-postgres-ca-cert
  2. Run from Postgres-operator/deploy: 
    ./configmap.sh
  3. Run to verify both are regurgitated (isc-cases-pgcluster-configmap and isc-cases-postgres-ca-cert): 
    oc get cm
  4. On a separate ssh session, perform: 
    oc port-forward svc/postgres-operator 8443:8443 -n <foundations namespace>
  5. On the original session, perform: 
    pgo delete cluster -d isc-cases-postgres -n <foundations namespace>
pgo create cluster isc-cases-postgres --custom-config isc-cases-pgcluster-configmap -n <foundations namespace>
  6. Check kubectl logs cases-applications for initialization or upgrade to ensure errors are resolved; steps outlines on crunchy data Postgres kb page:
    https://www.ibm.com/support/knowledgecenter/SSTDPP_1.1.0/docs/security-pak/postgrescerts.html

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m0z0000001h8pAAA","label":"Cases"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
02 May 2022

UID

ibm15691440

Page Feedback