Existing universal connector certificate does not work in a restored environment

Symptoms

After upgrading Guardium® Insights and restoring a backup, universal connector connections do not work, with a Filebeat log error similar to this:

{"log.level":"error","@timestamp":"2024-07-01T05:02:54.751-0700",
"log.logger":"publisher_pipeline_output",
"log.origin":{"function":"<location>/pipeline.(*netClientWorker).run",
"file.name":"pipeline/client_worker.go","file.line":148},
"message":"Failed to connect to backoff(async(tcp://<tcp>)): x509: 
certificate signed by unknown authority (possibly because of 
\"crypto/rsa: verification error\" while trying to verify candidate 
authority certificate \"ibm.com\")","service.name":"filebeat",
"ecs.version":"1.6.0"}

Causes

This is due to the new cluster having a different DNS or fully qualified domain name (FQDN).

Resolving the problem

Reconfigure the universal connector connections so that they match the new DNS or FQDN.

Confirm that the secret insights-ingressca holds a wild card certificate that contains a Subject Alternative Name (SAN) entry the same as hostName and *.domainName found in the Guardium Insights custom resource (CR) file. The paths for hostName and domainName in the CR are spec.guardiumInsightsGlobal.insights.ingress.hostName and spec.guardiumInsightsGlobal.insights.ingress.domainName respectively.

Run this command to check the SAN entries:

oc get secret insights-ingressca -o 'jsonpath={.data.tls\.crt}' | base64 -d | openssl x509 -text -noout | grep -B 1 'DNS'

This command will show the DNS endpoint for the SAN of the certificate. It should contain *.domainName, which means the certificate is a wild card certificate. For example:

DNS:*.staging.apps.ibm.com, 
DNS:*.staging.apps.ibm.com, 
DNS:staging.apps.ibm.com

If the certificate is not a wild card certificate, check the hostName and domainName specified in the Guardium Insights CR with this command:

oc get `oc get guardiuminsights -o name` -o 'jsonpath={.spec.guardiumInsightsGlobal.insights.ingress.hostName}'
oc get `oc get guardiuminsights -o name` -o 'jsonpath={.spec.guardiumInsightsGlobal.insights.ingress.domainName}'

The values for hostName and domainName must be the same. If they are not, edit them to be the same in the Guardium Insights CR with this command:

oc get guardiuminsights

If you are using your own certificate, and it is not a wild card certificate, the certificate needs to be replaced by following the steps in Replacing the ingress certificate.

If you want a self-signed certificate, delete the secret insights-ingressca (with oc delete secret insights-ingressca) and then the Guardium Insights operator will create a fresh certificate automatically. The new certificate will be a self-signed wild card certificate.

After replacing the certificate, update all agents that communicate with the Guardium Insights universal connector to use the new certificate.