Upgrade Issues

• tls errors found in the operator log

tls errors found in the operator log

After upgrading the IBM Storage Scale container native, you can see many messages in the operator log similar to the following:

http: TLS handshake error from <IP>:<PORT> remote error: tls: bad certificate

Verify that the Webhooks have a caBundle injected for each defined path: You can use these commands to check this:

• To check the MutatingWebhookConfiguration:

    oc get MutatingWebhookConfiguration ibm-spectrum-scale-mutating-webhook-configuration -oyaml | egrep "caBundle:|path:"
  

• To check the ValidatingWebhookConfiguration:

    oc get ValidatingWebhookConfiguration ibm-spectrum-scale-validating-webhook-configuration -oyaml | egrep "caBundle:|path:"
  

For example, if running the command against the ibm-spectrum-scale-mutating-webhook-configuration, you can see an output similar to the following:

$ oc get MutatingWebhookConfiguration ibm-spectrum-scale-mutating-webhook-configuration -oyaml | egrep "caBundle:|path:"
caBundle: <SOME CA BUNDLE DEFINED, LONG KEY>
  path: /mutate-scale-spectrum-ibm-com-v1beta1-cluster
  path: /mutate-scale-spectrum-ibm-com-v1beta1-filesystem

The output shows one caBundle: entry and two path: entries, meaning for the second path: /mutate-scale-spectrum-ibm-com-v1beta1-filesystem, a valid caBundle was NOT injected. This is caused by a bug in the Red Hat OpenShift service CA operator, OCPBUGS-8512.

If this has occurred in your cluster, resolve the problem by using the following steps:

1. Delete the MutatingWebhookConfiguration and ValidatingWebhookConfiguration.

    oc delete MutatingWebhookConfiguration ibm-spectrum-scale-mutating-webhook-configuration
    oc delete ValidatingWebhookConfiguration ibm-spectrum-scale-validating-webhook-configuration
   

2. Reapply the manifest for IBM Storage Scale container native. For example, if you are upgrading to v5.1.9, then run:

    oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.1.x/generated/scale/install.yaml
   

3. Running the webhook queries again should result in each endpoint having a caBundle injected in each path:

    caBundle: <SOME CA BUNDLE DEFINED, LONG KEY>
      path: /mutate-scale-spectrum-ibm-com-v1beta1-cluster
    caBundle: <SOME CA BUNDLE DEFINED, LONG KEY>
      path: /mutate-scale-spectrum-ibm-com-v1beta1-filesystem
   

For more information, see Upgrading IBM Storage Scale container native.