Upgrading IBM Cloud Private

You can upgrade IBM® Cloud Private from specific previous versions.

Supported upgrade paths

You can upgrade only the following supported paths:

IBM Cloud Private version 3.1.1 to 3.1.2
IBM Cloud Private version 3.1.0 to 3.1.2

If you use an earlier version of IBM Cloud Private, you must upgrade to version 3.1.0 first.

When you upgrade from version 3.1.0 and higher to 3.1.2 in a high availability (HA) IBM Cloud Private cluster, application pods continue to run during the upgrade. For details on an HA cluster, see High availability IBM Cloud Private clusters.

In general, traffic to applications continues to be routed even as management components are upgraded. During the upgrade, a brief outage can occur during the following scenarios:

The kube-dns upgrade.
When an external load balancer is used to route requests to the cluster ingress, the load balancer health check might fail. This failure interrupts incoming request traffic over the ingress until the connection is re-established between the external load balancer and the ingress.

Note: Ensure that you review and verify that you meet the increased memory requirements. For more information, see Hardware requirements.

During the upgrade process, you cannot access the IBM Cloud Private management console. You also cannot set cloud provider options, such as configuring a vSphere cloud provider, or using NSX-T.

If your current IBM Cloud Private installation is configured to use the Key Management Service (KMS) plug-in for encryption, you can preserve your existing KMS configuration. For more information, see Retaining KMS configuration during upgrade.

Upgrading

Log in to the boot node as a user with root permissions. The boot node is usually your master node. For more information about node types, see Architecture. During installation, you specify the IP addresses for each node type.
Download the installation files for IBM Cloud Private. These files are available for download from the IBM Passport Advantage® website.
- For a Linux® x86_64 cluster, download the ibm-cloud-private-x86_64-3.1.2.tar.gz file.
- For a Linux® on Power® (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.1.2.tar.gz file.
Extract the images and load them into Docker. Extracting the images might take a few minutes.
- For Linux x86_64, run the following command:
```
tar xf ibm-cloud-private-x86_64-3.1.2.tar.gz -O | sudo docker load
```
- For Linux on Power (ppc64le), run the following command:
```
tar xf ibm-cloud-private-ppc64le-3.1.2.tar.gz -O | sudo docker load
```
Create an installation directory and copy the cluster directories from the previous installation directory to the new IBM Cloud Private cluster folder. Use a different installation directory than you used for the previous version. For example, to store the configuration files in /opt/ibm-cloud-private-3.1.2, run the following commands:
```
 sudo mkdir -p /opt/ibm-cloud-private-3.1.2
 cd /opt/ibm-cloud-private-3.1.2
 sudo cp -r /<installation_directory>/cluster .
 sudo rm -rf cluster/.upgrade cluster/upgrade_version
```
Note: /<installation_directory> is the full path to your version 3.1.1 installation directory, and /<new_installation_directory> is the full path to your version 3.1.2 installation directory. It is not required to copy the entire image installation package from the previous version.
Check the calico_ipip_enabled parameter value in the version that you are upgrading from.
- If the parameter was set as calico_ipip_enabled: true, replace the parameter in the /<new_installation_directory>/cluster/config.yaml with calico_ipip_mode: Always.
- If the parameter was set as calico_ipip_enabled: false, replace the parameter in the /<new_installation_directory>/cluster/config.yaml with calico_ipip_mode: Never.
Move the image files for your cluster to the /<new_installation_directory>/cluster/images folder.
- For Linux x86_64, run this command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-x86_64-3.1.2.tar.gz cluster/images/
```
- For Linux on Power (ppc64le), run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-ppc64le-3.1.2.tar.gz cluster/images/
```
  If you have IBM® Z worker nodes in your cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-s390x-3.1.2.tar.gz cluster/images/
```
In this command, path_to_images_file is the path to the images file.
Deploy your environment by completing the following steps:
1. Change to the cluster folder in your installation directory.
```
cd /<new_installation_directory>/cluster
```
2. Prepare the cluster for upgrade:
  - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.1.2-ee upgrade-prepare
```
  - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.1.2-ee upgrade-prepare
```
  If the cluster preparation fails, review the error message and resolve any issues. Then, remove the cluster/.install.lock file, and run the upgrade-prepare command again.
3. Upgrade Kubernetes:
  - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.1.2-ee upgrade-k8s
```
  - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.1.2-ee upgrade-k8s
```
  - If the Kubernetes upgrade fails with a different message, review the error message and resolve any issues. Then, re-run the upgrade Kubernetes services command again.
4. Upgrade the charts:
  - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.1.2-ee upgrade-chart
```
  - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.1.2-ee upgrade-chart
```
  - If the chart upgrade fails with a different message, review the error message and resolve any issues. Then, re-run the upgrade chart command again.
5. If GlusterFS is installed in your cluster, you must upgrade the GlusterFS client to version 4.1.5.
Verify the status of your upgrade.
- If the upgrade succeeded, the access information for your cluster is displayed.
```
UI URL is https://<Cluster Master Host>:<Cluster Master API Port>
```
  Where, <Cluster Master Host>:<Cluster Master API Port> is defined in Master endpoint.
- If you encounter errors, see Troubleshooting.
Clear your browser cache.
If you have either applications that use GPU resources or a resource quota for GPU resources, you need to manually update the application or resource quota with the new GPU resource name nvidia.com/gpu.
- For applications that use GPU resources, follow the steps in Creating a deployment with attached GPU resources to run a sample GPU application. For your own GPU application, you need to update the application to use the new GPU resource name nvidia.com/gpu. For example, to update the deployment properties, you can use either the management console (see Modifying a deployment) or the kubectl CLI.
- To update the resource quota for GPU resources, follow the steps in Setting resource quota to set a resource quota for your namespace. For upgrading, you need to update the resource quota to use the GPU resource name nvidia.com/gpu. For example, you can set the GPU quota to requests.nvidia.com/gpu: "2".
Access your cluster. From a web browser, browse to the URL for your cluster. For a list of supported browsers, see Supported browsers.
- For more information about accessing your cluster by using the IBM Cloud Private management console from a web browser, see Accessing your IBM Cloud Private cluster by using the management console.
- For more information about accessing your cluster by using the Kubernetes command line (kubectl), see Accessing your IBM Cloud Private cluster by using the kubectl CLI. Note: After your upgrade the pod security policy for your clusters is automatically enabled, but set to the least restrictive setting to avoid access problems. See Pod security for information about how to manage the pod security policy settings.
Ensure that all the IBM Cloud Private default ports are open. For more information about the default IBM Cloud Private ports, see Default ports.
Back up the boot node. Copy your /<new_installation_directory>/cluster directory to a secure location.
If you use Cloud Automation Manager in your IBM Cloud Private cluster, you must also upgrade it. See Upgrading Cloud Automation Manager.