Upgrade from foundational services version 3.x to version 4.x fails due to custom certificate issue

If you used custom certificates in foundational services version 3, upgrade of foundational services to version 4.x fails.

Symptom

The custom certificates are not available in the cluster after the upgrade. The cp-console route then uses the default ingress certificates.

You see the following error after upgrade.

======>>> START-14 : update secrets 'custom-tls-secret'
Starting new HTTPS connection (1): kubernetes.default:443
https://kubernetes.default:443 "GET /api/v1/namespaces/aiops/secrets/custom-tls-secret HTTP/1.1" 404 208
Unable to GET URL :https://kubernetes.default:443/api/v1/namespaces/aiops/secrets/custom-tls-secret
<Response [404]>
FAILED : GET oper, URL : /api/v1/namespaces/aiops/secrets/custom-tls-secret

Cause

In the cs-onprem-tenant-config configmap, the custom_host_certificate_secret field uses a default name custom-tls-secret. If you use any other name for the secret, the upgrade fails.

Workaround

After you upgrade, complete the steps to create a custom TLS secret with the name custom-tls-secret. For more information, see Updating custom hostname and TLS secret by using a configmap.

Then, restart the IM and Zen pods.

Get the IM authentication pod name and delete the pod.

oc get pod -n <your-foundational-services-namespace> | grep platform-auth-service

Delete the auth pod.

oc delete pod -n <your-foundational-services-namespace> <platform-auth-service-pod-name>

Get the IM identity provider pod name and delete the pod.

oc get pod -n <your-foundational-services-namespace> | grep platform-identity-provider

Delete the identity provider pod.

oc delete pod -n <your-foundational-services-namespace> <platform-identity-provider-pod-name>

Get the Zen operator pod name and delete the pod.

oc get pod -n <your-foundational-services-namespace> | grep ibm-zen-operator

Delete the Zen operator pod.

oc delete pod -n <your-foundational-services-namespace> <ibm-zen-operator-pod-name>