IBM Support

Cloud Pak for Security: Web console login error upstream connect error or disconnect reset before headers

Troubleshooting


Problem

Cloud Pak for Security Web UI login fails with error:
"upstream connect error or disconnect/reset before headers. reset reason: connection failure".

Symptom

Evidence of messages for an expired certificate can be found in the logs of the following pods:
  • Pod clx-shell:
    {"level":"error","ibm_datetime":"YYYY-MM-DDTHH:MM:SS.NNNZ","pid":1,"hostname":"clx-shell-NNNNNNNNNN-NNNNN","label":"app.shell.connectredis","code":"CERT_HAS_EXPIRED","stack":"Error: certificate has expired
    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket._finishInit (_tls_wrap.js:932:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)","type":"Error","message":"certificate has expired"}
  • Pod authsvc:
    {"level":"error","ibm_datetime":"YYYY-MM-DDTHH:MM:SS.NNNZ","caller":"auth/context.go:57","log":"Failed to retrieve APIKey","req.method":"GET","req.url":"/api/configstore/v1/config/uds-ds-connections/NNNNNNNN-NNNN-NNNN-NNNN-NNNNNNNNNNNN","apikey":"<KEY>","res.statusCode":503,"error":"Get \"https://default-couchdbcluster.cp4s.svc.cluster.local/apikeys/<KEY>\": x509: certificate has expired or is not yet valid: current time YYYY-MM-DDTHH:MM:SSZ is after YYYY-MM-DDTHH:MM:SSZ","stacktrace":"github.ibm.com/security-secops/isc-common-authbridge/auth/pkg/auth.(*Context).Log\n\t/workspace/pkg/auth/context.go:57\ngithub.ibm.com/security-secops/isc-common-authbridge/auth/pkg/w3.AuthAll\n\t/workspace/pkg/w3/all.go:23\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2042\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2417\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2843\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1925"}
  • Pod isc-entitlements:
    DATABASE NAME: icp-entitlements
    VIEWS DIRECTORY: /entitlements/views
    Failed to update views
    'Error: certificate has expired\n' +
    {"level":"error","label":"redis-client","error":"An error has occurred with Redis: Error: certificate has expired","ibm_datetime":"YYYY-MM-DDTHH:MM:SS.NNNZ","message":"[]"}

Cause

The certificate for the couch database is expired.

Resolving The Problem

Run the following script:
for POD in $(oc get pod --no-headers | awk '/^c-default/ {print $1}'); do { oc delete pod $POD; } & done

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m3p0000000rbnAAA","label":"Administration Task"}],"ARM Case Number":"TS008236592","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"1.7.2"}]

Document Information

Modified date:
03 November 2022

UID

ibm16551388