Applying fix packs to your cluster

After you install IBM Cloud Private, you can check IBM® Fix Central to see whether fix packs that need to be applied to your cluster are available.

IBM® Fix Central contains fixes and updates for the product. To access this website, see IBM® Fix Central Opens in a new tab .

You can also subscribe to notifications to get updates through email or RSS feed on the newest security bulletins, known issues, and fix packs that are being released. For more information, see My Notifications Opens in a new tab .

If a fix pack is available for your cluster, download and apply the fix pack.

Supported fix pack upgrade paths
Downloading a fix pack
Applying a fix pack
Verifying that a fix pack is applied
Rolling back a fix pack

When you apply a fix pack, changes are applied to only the management services that are enabled. If a management service is not available, any fix for that service is skipped during the process for applying the fix pack. If you want to enable a management service and apply any fixes for that service, you need to directly upgrade the Helm release for the service. For more information about how to upgrade the Helm release, see Managing Helm releases.

When a fix is applied to an enabled management service, the configurations for that service can be overwritten. The process to apply a fix pack uses the config.yaml to enforce consistency with the mapping of enabled and disabled services. This behavior can result in charts that you manually deployed that use the reserved name for the service, and the deploying charts to be deleted. This process can also overwrite any deployments that used the reserved name. Any values that you configured within charts that you deployed with the installer are persisted. If you want to keep your existing configurations, complete either of the following tasks:

Update the config.yaml to represent the configuration for the deployed chart for the service.
Disable the service within the config.yaml while you are applying the fix pack so that the service is skipped during the application process. After the fix pack is applied, enable the service again.

Important:

You can apply a fix pack only on IBM Cloud Private 3.2.1 environments or environments that are on a newer fix pack version of 3.2.1. If you are not currently on IBM Cloud Private 3.2.1 or a newer fix pack version, you can install or upgrade directly to an IBM Cloud Private 3.2.1 fix pack version. For more information on installing or upgrading to a fix pack version, see:
- Installing
- Upgrading
Currently, there are two fix pack versions available, 3.2.1.x fix packs and 3.2.2.x fix packs. The 3.2.1.x fix packs are intended for environments that include Kubernetes version 1.13.12. The 3.2.2.x fix packs include fixes to upgrade the supported version of Kubernetes. The fixes that are included within these 3.2.2.x fix packs include all fixes that are included within the equivalent 3.2.1.x fix pack, except for Kubernetes specific fixes. If you apply a 3.2.2.x fix pack, do not apply an equivalent 3.2.1.x fix pack.

The latest 3.2.1.x fix pack is 3.2.1.2203. The latest 3.2.2.x fix pack is 3.2.2.2203 and upgrades Kubernetes to version 1.19.3.
If vip_manager: ucarp is set in your existing cluster, and if you are upgrading to the 3.2.2.2008 fix pack, you must change the setting to vip_manager: keepalived in the config.yaml before you upgrade. The ucarp support is removed.

Supported fix pack upgrade paths

The following table shows the upgrade paths: | Version you are upgrading from | Version you are upgrading to | Procedure | |---|---|---| | 3.2.2.2105 | 3.2.2.2203 | Applying fix pack| | 3.2.2.2012 | 3.2.2.2105 or 3.2.2.2203 | Applying fix pack| | 3.2.2.2008 | 3.2.2.2012 or 3.2.2.2105 or 3.2.2.2203 | Upgrade | | 3.2.2.2006 | 3.2.2.2008 | Applying a fix pack| | 3.2.1.2003, 3.2.1.2006, or 3.2.1.2008| 3.2.2.2008 | Applying a fix pack | | 3.2.1.x fix pack prior to 3.2.1.2003, 3.2.1 version | 3.2.1.2003 | Applying a fix pack |

Apply the fix pack to IBM Cloud Private 3.2.2.x fix pack version

You can upgrade to the 3.2.2.2105 or 3.2.2.2203 fix pack from only a 3.2.2.2008 fix pack version.

If you are on the 3.2.2.2003 fix pack version, you must first apply the 3.2.2.2008 fix pack before applying the 3.2.2.2012 or 3.2.2.2105, or 3.2.2.2203 fix pack. From the 3.2.2.2008 fix pack, you can apply the 3.2.2.2105 or 3.2.2.2203 fix pack.

If you are on the 3.2.2.2012 fix pack version, you can apply the 3.2.2.2105 or 3.2.2.2203 fix pack.

For detailed instructions to apply a fix pack other than the 3.2.2.2105 fix pack, see Applying fix packs to your cluster.

If you do want to apply, or upgrade to, a 3.2.2.x fix pack, you must first install or upgrade to the 3.2.1.2003 or newer 3.2.1.x fix pack. After you apply the 3.2.1.2003 or newer 3.2.1.x fix pack version, you can repeat the same steps and apply the 3.2.2.2006 or 3.2.2.2008 fix pack to upgrade Kubernetes to version 1.16.7.

For downloading and upgrading to the 3.2.2.2203 fix pack from the 3.2.2.2008 or 3.2.2.2006 fix pack, you need to follow the procedure for upgrading to IBM Cloud Private 3.2.1, but use the package and commands to upgrade to 3.2.2.2203. For more information, see Upgrading.

Downloading a fix pack

You can download available fix packs for IBM Cloud Private from IBM® Fix Central.

Log in to IBM® Fix Central .
Search for and download the appropriate fix pack for your environment:

For downloading the 3.2.2.2105 fix pack:
- For a Linux® x86_64 cluster, download the ibm-cloud-private-x86_64-3.2.2.2105.tar.gz file.
- For a Linux® on Power® (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.2.2.2105.tar.gz file.
- For a IBM® Z cluster, download the ibm-cloud-private-s390x-3.2.2.2105.tar.gz file.
- For a IBM Cloud Private with OpenShift cluster, download the ibm-cloud-private-rhos-3.2.2.2105.tar.gz file.
For downloading and upgrading to the 3.2.2.2203 fix pack from the 3.2.2.2008 or 3.2.2.2006 fix pack, see Upgrading.

For downloading the 3.2.1.2203 fix pack:
- For a Linux x86_64 cluster, download the ibm-cloud-private-x86_64-3.2.1.2203.tar.gz file.
- For a Linux on Power (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.2.1.2203.tar.gz file.
- For a Linux on IBM Z and LinuxONE cluster, download the ibm-cloud-private-s390x-3.2.1.2203.tar.gz file.
- For a IBM Cloud Private with OpenShift cluster, download the ibm-cloud-private-rhos-3.2.1.2203.tar.gz file.

Applying a fix pack

You can apply an IBM Cloud Private fix pack to your cluster from specific versions of IBM Cloud Private.

Note: The apply fix pack feature, apply-fixpack, and the following steps are supported for only a standard IBM Cloud Private installation and IBM Cloud Private with OpenShift. These steps are not supported for IBM Cloud Private with IBM Kubernetes Service, or IBM Cloud Private with klusterlet.

Log in to the boot node as a user with root permissions. The boot node is usually your master node.
Download the fix pack file.
Extract the images from the fix pack and load the images into Docker. Extracting the images can take a few minutes.

These commands use the 3.2.2.2105 fix pack as an example. You can only apply 3.2.2.2105 fix pack on 3.2.2.2012. If you are applying a different fix pack, such as the 3.2.1.2105 fix pack, replace 3.2.2.2105 with 3.2.1.2105 in the command that you run.
- For a Linux x86_64 cluster, run the following command:
```
tar xf ibm-cloud-private-x86_64-3.2.2.2105.tar.gz -O | sudo docker load
```
- For a Linux on Power (ppc64le) cluster, run the following command:
```
tar xf ibm-cloud-private-ppc64le-3.2.2.2105.tar.gz -O | sudo docker load
```
- For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
tar xf ibm-cloud-private-s390x-3.2.2.2105.tar.gz -O | sudo docker load
```
- For a IBM Cloud Private with OpenShift cluster, run the following command:
```
tar xf ibm-cloud-private-rhos-3.2.2.2105.tar.gz -O | sudo docker load
```
Move the fix pack file to your cluster /<installation_directory>/cluster/images folder.
1. From the command line, change directories to the cluster folder in your installation directory.
```
  cd /<installation_directory>/cluster
```
2. Move the fix pack file.
  
  These commands use the 3.2.2.2105 fix pack as an example. You can only apply 3.2.2.2105 fix pack on 3.2.2.2012. If you are applying a different fix pack, such as the 3.2.1.2105 fix pack, replace 3.2.2.2105 with 3.2.1.2105 in the command that you run.
  - For a Linux x86_64 cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-x86_64-3.2.2.2105.tar.gz images/
```
  - For a Linux on Power (ppc64le) cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-ppc64le-3.2.2.2105.tar.gz images/
```
  - For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-s390x-3.2.2.2105.tar.gz images/
```
  - For a IBM Cloud Private with OpenShift cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-rhos-3.2.2.2105.tar.gz images/
```
  The value for <path_to_images_file> is the path to the fix pack package.
Update the Configmap addon-deploy-status for all charts.
1. Backup the addon-deploy-status file by running the following command:
```
 kubectl -n kube-system get cm addon-deploy-status -o yaml > addon-deploy-status.bak.yaml
```
2. Remove the addon-deploy-status file by running the following command:
```
 kubectl -n kube-system delete cm addon-deploy-status
```

Apply the fix pack images to your cluster. Run the following command:

These commands use the 3.2.2.2105 fix pack as an example. You can only apply 3.2.2.2105 fix pack on 3.2.2.2012. If you are applying a different fix pack, such as the 3.2.1.2105 fix pack, replace 3.2.2.2105 with 3.2.1.2105 in the command that you run.

For a Linux x86_64 cluster, run the following command:

sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.2.2105-ee apply-fixpack

For a Linux on Power (ppc64le) cluster, run the following command:

sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.2.2105-ee apply-fixpack

For a Linux on IBM Z and LinuxONE cluster, run the following command:

sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.2.2105-ee apply-fixpack

For a IBM Cloud Private with OpenShift cluster, run the following command:

docker run -e LICENSE=accept --rm -v $(pwd):/installer/cluster:z -v /var/run:/var/run:z --security-opt label:disable ibmcom/icp-inception-amd64:3.2.2.2105-rhel-ee apply-fixpack

After you apply fix pack version 3.2.1.2012 or newer 3.2.1.x fix pack, add the root CA certificate to your truststore. With this fix pack, users on macOS 10.15 or newer cannot access the management console until the root CA certificate is added to the truststore. For more information, see:
- Refreshing root CA certificates
- Troubleshooting console access on macOS Catalina

For fix pack versions 3.2.1.2001 and and 3.2.1.2003, after you apply the fix pack, check the cert-manager-webhook logs for potential errors. Run the following command to check the logs:

kubectl logs -n cert-manager $(kubectl get pods -n cert-manager -o custom-columns=NAME:metadata.name --no-headers | grep webhook | grep -v cainjector)

Scan the logs for any error lines that look similar to the following errors:

E0207 21:03:57.004338       1 errors.go:77] Post https://10.0.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: http2: no cached connection was available
E0207 21:03:57.289384       1 webhook.go:192] Failed to make webhook authorizer request: Post https://10.0.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews: http2: no cached connection was available

If you see any errors, run the following command to restart the webhook pod:

kubectl delete pod -n cert-manager $(kubectl get pods -n cert-manager -o custom-columns=NAME:metadata.name --no-headers | grep webhook | grep -v cainjector)

Check the logs again to ensure that the errors no longer exist:

kubectl logs -n cert-manager $(kubectl get pods -n cert-manager -o custom-columns=NAME:metadata.name --no-headers | grep webhook | grep -v cainjector)

If you applied the 3.2.2.2008 fix pack or previous 3.2.2.x fix pack, which includes an updated version of Kubernetes, you might need to complete some additional steps:
- If your environment includes an application, the application deployables might now have a Failed status. To resolve this issue, restart any appmgr pods (multicluster-endpoint/endpoint-appmgr-<string>) pods on the managed clusters that include failed deployables to refresh the status.
- After you upgrade to the fix pack, the vulnerability-advisor-cos-indexer pod might be in CrashLoopBackOff status. This status can occur due to an inconsistent pod restart of Kafka and MinIO. To resolve this issue and refresh the pod status, restart the following pods:
  - vulnerability-advisor-cos-indexer
  - vulnerability-advisor-minio
  - vulnerability-advisor-kafka
  To restart and reconfigure the pods, run the following command for each pod. Replace pod_name with the name of the pod to delete and restart.
```
kubectl delete pod pod_name -n kube-system
```

Verifying that the fix pack is applied

If the fix pack is successfully applied to your cluster, the output for the apply-fixpack command shows the access information for your cluster:

The Dashboard URL: https://<Cluster Master Host>:<Cluster Master API Port>

The <Cluster Master Host>:<Cluster Master API Port> value is defined in the Master endpoint. For more information, see IBM Cloud Private endpoints.

Rolling back a fix pack

If you encounter issues due to this fix pack, troubleshoot the issue and reapply this fix pack. If needed, you can roll back the fix pack changes to specific previous versions. For more information, see Rolling back a fix pack.

Upgrading the multicluster endpoint operator

If you are applying a fix pack to a managed cluster where the multicluster endpoint operator is installed, upgrade the operator to the fix pack version. For more information, see Upgrading the multicluster endpoint operator to a fix pack version.