Applying fix packs to your cluster

After you install IBM Cloud Private, you can check IBM® Fix Central to see whether fix packs that need to be applied to your cluster are available.

IBM® Fix Central contains fixes and updates for the product. To access this website, see IBM® Fix Central Opens in a new tab .

You can also subscribe to notifications to get updates through email or RSS feed on the newest security bulletins, known issues, and fix packs that are being released. For more information, see My Notifications Opens in a new tab .

If a fix pack is available for your cluster, download and apply the fix pack.

Downloading a fix pack
Applying a fix pack
Verifying that a fix pack is applied
Rolling back a fix pack

Note: When you apply a fix pack, changes are applied to only the management services that are enabled. If a management service is not available, any fix for that service is skipped during the process for applying the fix pack. If you want to enable a management service and apply any fixes for that service, you need to directly upgrade the Helm release for the service. For more information about how to upgrade the Helm release, see Managing Helm releases.

When a fix is applied to an enabled management service, the configurations for that service can be overwritten. The process to apply a fix pack uses the config.yaml to enforce consistency with the mapping of enabled and disabled services. This behavior can result in charts that you manually deployed that use the reserved name for the service, and the deploying charts to be deleted. This process can also overwrite any deployments that used the reserved name. Any values that you configured within charts that you deployed with the installer are persisted. If you want to keep your existing configurations, complete either of the following tasks:

Update the config.yaml to represent the configuration for the deployed chart for the service.
Disable the service within the config.yaml while you are applying the fix pack so that the service is skipped during the application process. After the fix pack is applied, enable the service again.

Important: If vip_manager: ucarp is set in your existing cluster, and if you are upgrading to the 3.2.0.2003 fix pack, you must change the setting to vip_manager: keepalived in the config.yaml before you upgrade. The ucarp support is removed.

Downloading a fix pack

You can download available fix packs for IBM Cloud Private from IBM® Fix Central.

Log in to IBM® Fix Central .
Search for and download the appropriate fix pack for your environment:
- For a Linux x86_64 cluster, download the ibm-cloud-private-x86_64-3.2.0.2003.tar.gz file.
- For a Linux on Power (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.2.0.2003.tar.gz file.
- For a Linux on IBM Z and LinuxONE cluster, download the ibm-cloud-private-s390x-3.2.0.2003.tar.gz file.
- For a IBM Cloud Private with OpenShift cluster, download the ibm-cloud-private-rhos-3.2.0.2003.tar.gz file.

Applying a fix pack

You can apply an IBM Cloud Private fix pack to your cluster from specific versions of IBM Cloud Private.

Note: The apply fix pack feature, apply-fixpack, and the following steps are supported for only a standard IBM Cloud Private installation and IBM Cloud Private with OpenShift. These steps are not supported for IBM Cloud Private with IBM Kubernetes Service, or IBM Cloud Private with klusterlet.

Log in to the boot node as a user with root permissions. The boot node is usually your master node.
Download the fix pack file.
Extract the images from the fix pack and load the images into Docker. Extracting the images can take a few minutes.
- For a Linux x86_64 cluster, run the following command:
```
tar xf ibm-cloud-private-x86_64-3.2.0.2003.tar.gz -O | sudo docker load
```
- For a Linux on Power (ppc64le) cluster, run the following command:
```
tar xf ibm-cloud-private-ppc64le-3.2.0.2003.tar.gz -O | sudo docker load
```
- For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
tar xf ibm-cloud-private-s390x-3.2.0.2003.tar.gz -O | sudo docker load
```
- For a IBM Cloud Private with OpenShift cluster, run the following command:
```
tar xf ibm-cloud-private-rhos-3.2.0.2003.tar.gz -O | sudo docker load
```
Move the fix pack file to your cluster /<installation_directory>/cluster/images folder.
1. From the command line, change directories to the cluster folder in your installation directory.
```
  cd /<installation_directory>/cluster
```
2. Move the fix pack file.
  - For a Linux x86_64 cluster, run the following command:
```
  sudo mv /<path_to_images_file>/ibm-cloud-private-x86_64-3.2.0.2003.tar.gz images/
```
  - For a Linux on Power (ppc64le) cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-ppc64le-3.2.0.2003.tar.gz images/
```
  - For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-s390x-3.2.0.2003.tar.gz images/
```
  - For a IBM Cloud Private with OpenShift cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-rhos-3.2.0.2003.tar.gz images/
```
  The value for <path_to_images_file> is the path to the fix pack package.
Apply the fix pack images to your cluster. Run the following command:
- For a Linux x86_64 cluster, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0.2003-ee apply-fixpack
```
- For a Linux on Power (ppc64le) cluster, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0.2003-ee apply-fixpack
```
- For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0.2003-ee apply-fixpack
```
- For a IBM Cloud Private with OpenShift cluster, run the following command:
  - If your boot node is a dedicated boot node, use this command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0.0-rhel-ee apply-fixpack
```
  - If your boot node is an OpenShift master node, use this command:
```
 docker run -e LICENSE=accept --rm -v $(pwd):/installer/cluster:z -v /var/run:/var/run:z --security-opt label:disable ibmcom/icp-inception-amd64:3.2.0.2003-rhel-ee apply-fixpack
```
  Note: If you encounter an error when applying the fix pack that results in a failure to deploy a chart, you must delete the job for deploying that chart. Then, run the command to apply the fix pack again. For example, if you encounter the error and deploying the multicluster-hub-ibm-mcm-prod-redis-secret-job chart fails, the job for that chart blocks the process to apply the fix pack from completing. You can run the following command to delete the job:
```
kubectl delete job multicluster-hub-ibm-mcm-prod-redis-secret-job -n kube-system --ignore-not-found
```
After you apply fix pack version 3.2.0.2003, add the root CA certificate to your trust store. With this fix pack, users on macOS 10.15 or newer cannot access the management console until the root CA certificate is added to the trust store. For more information, see:
- Refreshing root CA certificates
- Troubleshooting console access on macOS Catalina

Verifying that the fix pack is applied

If the fix pack is successfully applied to your cluster, the output for the apply-fixpack command shows the access information for your cluster:

The Dashboard URL: https://<Cluster Master Host>:<Cluster Master API Port>

The <Cluster Master Host>:<Cluster Master API Port> value is defined in the Master endpoint. For more information, see IBM Cloud Private endpoints.

Rolling back a fix pack

If you encounter issues due to this fix pack, troubleshoot the issue and reapply this fix pack. If needed, you can roll back the fix pack changes to specific previous versions. The following rollback scenarios are supported:

Roll back an applied fix pack to a previous fix pack level for IBM Cloud Private 3.2.0.
Roll back an applied fix pack to IBM Cloud Private 3.2.0.

Note: IBM Cloud Private 3.2.0 does not include the apply fix pack feature, apply-fixpack. This feature is used for both applying and rolling back a fix pack. If you want to roll back from a fix pack to IBM Cloud Private 3.2.0, you need to use the installer image ibmcom/icp-inception-amd64:3.2.0.0-ee to use the feature. This installer image is included within the IBM Cloud Private installation package. Include this installation package within your /<installation_directory>/cluster/images folder so that the command for applying or rolling back a fix is available. For more information, see Loading the fix pack apply feature.
Roll back an applied fix pack to a previously installed IBM Cloud Private version 3.1.0, 3.1.1, or 3.1.2. For more information, see Reverting to an earlier version of IBM Cloud Private.

To roll back a fix pack to IBM Cloud Private 3.2.0 or a previous fix pack level for IBM Cloud Private 3.2.0, complete the following steps:

Log in to the boot node as a user with root permissions. The boot node is usually your master node.
The platform-header-kubectl job, which is a pre-rollback hook of the platform-ui chart, blocks the rollback. When the platform-ui is rolling back, a new platform-header-kubectl job needs to be generated. You must delete any existing platform-header-kubectl job in the cluster before you attempt to roll back the fix pack again. Run the following command to delete the job:
```
kubectl delete job platform-header-kubectl -n kube-system --ignore-not-found
```

Run the commands to roll back a fix pack. The following step rolls back to IBM Cloud Private 3.2.0.2001.

For a Linux x86_64 cluster, run the following command:

sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0.2001-ee apply-fixpack

For a Linux on Power (ppc64le) cluster, run the following command:

sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0.2001-ee apply-fixpack

For a Linux on IBM Z and LinuxONE cluster, run the following command:

sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0.2001-ee apply-fixpack

For a IBM Cloud Private with OpenShift cluster:
1. Run the following command to add the role=master label to the master node. This label must be added to your master node before you run the command to roll back the fix pack, otherwise the rollback can fail.
```
kubectl label --overwrite node <icp-master-node> role=master.
```
2. Run the apply-fixpack command to roll back the fix pack:
  - If your boot node is a dedicated boot node, use this command:
```
 sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
 icp-inception-amd64:3.2.0.2001-rhel-ee apply-fixpack
```
  - If your boot node is an OpenShift master node, use this command:
```
 docker run -e LICENSE=accept --rm -v $(pwd):/installer/cluster:z -v /var/run:/var/run:z --security-opt label:disable ibmcom/icp-inception-amd64:3.2.0.2001-rhel-ee apply-fixpack
```

Verify the status of the rollback. If the rollback is successful, the command output for the apply-fixpack command shows the access information for your cluster:
```
   UI URL is https://<Cluster Master Host>:<Cluster Master API Port>
```
The <Cluster Master Host>:<Cluster Master API Port> value is defined in the Master endpoint.
Optional. Uninstall any helm charts that are introduced in the fix pack. If a helm chart is introduced in the fix pack or newer version of IBM Cloud Private that you are rolling back to, you cannot roll back the helm chart. Instead, you need to directly uninstall the helm chart.

To uninstall a helm chart, run the following command:
```
   cloudctl login -a https://<Cluster Master Host>:<Cluster Master API Port>
   helm delete --purge <helm chart> --tls
   cloudctl logout
```
The value for <helm chart> is the name of the helm chart that you are uninstalling.

Loading the fix pack apply feature

If you want to roll back to IBM Cloud Private 3.2.0, you need to first load support for the fix pack apply feature, apply-fixpack, to your cluster. To roll back from a fix pack to IBM Cloud Private 3.2.0, you need to use the installer image ibmcom/icp-inception-amd64:3.2.0.0-ee to use the feature. This installer image is included within the IBM Cloud Private installation package, ibm-cloud-private-<arch>-3.2.0.0.tar.gz package. Include this installation package within your /<installation_directory>/cluster/images folder so that the command for applying or rolling back a fix is available.

Log in to IBM® Fix Central .
Search for and download the installation file or image for the type of nodes in your cluster
- For a Linux x86_64 cluster, download the ibm-cloud-private-x86_64-3.2.0.0.tar.gz file.
- For a Linux on Power (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.2.0.0.tar.gz file.
- For a Linux on IBM Z and LinuxONE cluster, download the ibm-cloud-private-s390x-3.2.0.0.tar.gz file.
- For a IBM Cloud Private with OpenShift cluster, download the ibm-cloud-private-rhos-3.2.0.0.tar.gz file.
Extract the images from the load the images into Docker. Extracting the images can take a few minutes.
- For a Linux x86_64, run this command:
```
 tar xf ibm-cloud-private-x86_64-3.2.0.0.tar.gz -O | sudo docker load
```
- For a Linux on Power (ppc64le), run this command:
```
 tar xf ibm-cloud-private-ppc64le-3.2.0.0.tar.gz -O | sudo docker load
```
- For a Linux on IBM Z and LinuxONE, run this command:
```
 tar xf ibm-cloud-private-s390x-3.2.0.0.tar.gz -O | sudo docker load
```
- For a IBM Cloud Private with OpenShift cluster, run the following command:
```
 tar xf ibm-cloud-private-rhos-3.2.0.0.tar.gz -O | sudo docker load
```
Move the image files for your cluster to the /<installation_directory>/cluster/images folder.
1. From the command line, change directories to the cluster folder in your installation directory.
```
cd /<installation_directory>/cluster
```
2. If your cluster contains any x86_64 nodes, place the x86 package in the images directory. Add the path to your installation image file to the following command:
```
sudo mv ibm-cloud-private-x86_64-3.2.0.0.tar.gz cluster/images/
```
3. If your cluster contains any ppc64le nodes, place the ppc64le package in the images directory. Add the path to your installation image file to the following command:
```
sudo mv ibm-cloud-private-ppc64le-3.2.0.0.tar.gz cluster/images/
```
4. If your cluster contains any s390x nodes, place the s390x package in the images directory. Add the path to your installation image file to the following command:
```
sudo mv ibm-cloud-private-s390x-3.2.0.0.tar.gz cluster/images/
```