IBM Support

QRadar SOAR AppHost: Expired K3s certificates are not automatically rotated causing connection issues

Troubleshooting


Problem

Cached K3s certificates are not cleared when automatically rotated.
 
K3s generates internal certificates with a 1-year lifetime. Restarting the K3s service automatically rotates certificates that expired or are due to expire within 90 days. However, the version of K3s used with App Host does not clear out the cached certificate, which causes the same problem. Therefore, the cache needs to be cleared manually.
 
 

Symptom

Using the kubectl cli tool results in the following error:

 Unable to connect to the server: x509: certificate has expired or is not yet valid
 

Cause

The currently used version of K3s (v1.18) does not clear the cached certificate.  So even if the certificates are rotated by a K3s restart, the problem persists.

Diagnosing The Problem

 
Changes to an existing app that uses the SOAR platform does not succeed. This includes file changes, secret changes, deploy or undeploy, and upgrade requests.
 
Installing a new app, the status remains in a ‘Deploying’ state. A tooltip instructs the user to run sudo appHostPackageLogs. 
Running this command or any other command that starts with kubectl gives you the following error:
 
$ sudo kubectl get pods -A
Unable to connect to the server: x509: certificate has expired or is not yet valid
 
Running this command gives the following error:
$ sudo manageAppHost showconfig
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://127.0.0.1:6443/api/v1/namespaces?labelSelector=app.kubernetes.io%2Fname%3Dcontroller. Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized.
 
You can check the expiration data of a cached certificate by running the following command on the App Host server:
$ openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate
 
 
 

Resolving The Problem

As a precautionary measure backup the TLS dir.

sudo tar -czvf /var/lib/rancher/k3s/server/apphost-cert.tar.gz /var/lib/rancher/k3s/server/tls

Remove the following file.

sudo rm /var/lib/rancher/k3s/server/tls/dynamic-cert.json

Remove the cached certificate from a kubernetes secret.

sudo kubectl --insecure-skip-tls-verify=true delete secret -n kube-system k3s-serving

Restart the K3s service to rotate the certificates.

sudo systemctl restart k3s

Verify that kubectl commands function.

sudo kubectl get pods -A

Additionally, you can verify that all K3s internal certificates are no longer due to expire.

sudo su
for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done

Or run

curl -v -k https://localhost:6443 [https://localhost:6443] to confirm the new date of your app host cert
 

Document Location

Worldwide

[{"Type":"SW","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSIP9Q","label":"IBM Security SOAR"},"ARM Category":[{"code":"a8m0z0000001jTpAAI","label":"Integrations-\u003EAppHost"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]

Document Information

Modified date:
04 January 2022

UID

ibm16444205

Page Feedback