Upgrading IBM Cloud Private

You can upgrade IBM Cloud Private from specific previous versions.

Supported upgrade paths

You can upgrade only the following supported paths:

IBM Cloud Private version 3.1.2 to 3.2.0 or to 3.2.0.2003 (fix pack)
IBM Cloud Private version 3.1.1 to 3.2.0 or to 3.2.0.2003 (fix pack)
IBM Cloud Private version 3.1.0 to 3.2.0 or to 3.2.0.2003 (fix pack)

If you use an earlier version of IBM Cloud Private, you must upgrade to version 3.1.0 first.

Note: Ensure that you review and verify that you meet the increased memory requirements. For more information, see Hardware requirements.

During the upgrade process, you cannot access the IBM Cloud Private management console. You also cannot set cloud provider options, such as configuring a vSphere cloud provider, or use NSX-T.

If your current IBM Cloud Private installation is configured to use the Key Management Service (KMS) plug-in for encryption, you can preserve your existing KMS configuration. For more information, see Retaining KMS configuration during upgrade.

Upgrading

Log in to the boot node as a user with root permissions. The boot node is usually your master node. For more information about node types, see Architecture. During installation, you specify the IP addresses for each node type.
Download the installation files for your upgrade:
- If you are upgrading to IBM Cloud Private version 3.2.0, these files are available for download from the IBM Passport Advantage® website.
  - For a Linux® x86_64 cluster, download the ibm-cloud-private-x86_64-3.2.0.tar.gz file.
  - For a Linux® on Power® (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.2.0.tar.gz file.
  - For a IBM® Z cluster, download the ibm-cloud-private-s390x-3.2.0.tar.gz file.
- If you are upgrading to IBM Cloud Private fix pack version 3.2.0.2003, these files are available for download from the IBM® Fix Central website.
  - For a Linux® x86_64 cluster, download the ibm-cloud-private-x86_64-3.2.0.2003.tar.gz file.
  - For a Linux® on Power® (ppc64le) cluster, download the ibm-cloud-private-ppc64le-3.2.0.2003.tar.gz file.
  - For a IBM® Z cluster, download the ibm-cloud-private-s390x-3.2.0.2003.tar.gz file.
Extract the images and load them into Docker. Extracting the images might take a few minutes.
- If you are upgrading to IBM Cloud Private version 3.2.0, run the following command:
  - For a Linux x86_64 cluster, use the command:
```
tar xf ibm-cloud-private-x86_64-3.2.0.tar.gz -O | sudo docker load
```
  - For a Linux on Power (ppc64le) cluster, use the command:
```
tar xf ibm-cloud-private-ppc64le-3.2.0.tar.gz -O | sudo docker load
```
  - For a Linux on IBM Z and LinuxONE cluster, use the command:
```
tar xf ibm-cloud-private-s390x-3.2.0.tar.gz -O | sudo docker load
```
- If you are upgrading to IBM Cloud Private fix pack version 3.2.0.2003, run the following command:
  - For Linux x86_64, run the following command:
```
tar xf ibm-cloud-private-x86_64-3.2.0.2003.tar.gz -O | sudo docker load
```
  - For Linux on Power (ppc64le), run the following command:
```
tar xf ibm-cloud-private-ppc64le-3.2.0.2003.tar.gz -O | sudo docker load
```
  - For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
tar xf ibm-cloud-private-s390x-3.2.0.2003.tar.gz -O | sudo docker load
```
Create an installation directory and copy the cluster directories from the previous installation directory to the new IBM Cloud Private cluster folder. Use a different installation directory than you used for the previous version. For example, to store the configuration files in /opt/ibm-cloud-private-3.2.0, run the following commands:
```
 sudo mkdir -p /opt/ibm-cloud-private-3.2.0
 cd /opt/ibm-cloud-private-3.2.0
 sudo cp -r /<installation_directory>/cluster .
 sudo rm -rf cluster/.upgrade
```
Note: /<installation_directory> is the full path to your version 3.1.2 installation directory, and /<new_installation_directory> is the full path to your version 3.2.0 installation directory. It is not required to copy the entire image installation package from the previous version.
Check the calico_ipip_enabled parameter value in the version that you are upgrading from.
- If the parameter was set as calico_ipip_enabled: true, replace the parameter in the /<new_installation_directory>/cluster/config.yaml with calico_ipip_mode: Always.
- If the parameter was set as calico_ipip_enabled: false, replace the parameter in the /<new_installation_directory>/cluster/config.yaml with calico_ipip_mode: Never.
Move the image files for your cluster to the /<new_installation_directory>/cluster/images folder.
- For Linux x86_64, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-x86_64-3.2.0.tar.gz cluster/images/
```
- For Linux on Power (ppc64le), run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-ppc64le-3.2.0.tar.gz cluster/images/
```
- For a Linux on IBM Z and LinuxONE cluster, run the following command:
```
sudo mv /<path_to_images_file>/ibm-cloud-private-s390x-3.2.0.tar.gz cluster/images/
```
  In this command, path_to_images_file is the path to the images file.
Check the management_services section in your config.yaml file and disable multicluster-hub if you did not previously configure IBM Multicloud Manager. See the following example:
```
 management_services:
   multicluster-hub: disabled
```
Note: With IBM Multicloud Manager disabled, some of the Search features are also disabled.
Deploy your environment by completing the following steps:
1. If you enabled single sign-on (SSO) in a previous release, you must disable it before you upgrade your cluster. For more information about disabling SSO, see Configuring single sign-on. If you do not disable SSO before upgrade, or if the platform-auth container fails to start after you upgrade your cluster, see platform-auth container fails to start.
2. Change to the cluster folder in your installation directory.
```
cd /<new_installation_directory>/cluster
```
3. Prepare the cluster for upgrade:
  - If you are upgrading to IBM Cloud Private version 3.2.0, run the following command:
    - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0-ee upgrade-prepare
```
    - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0-ee upgrade-prepare
```
    - For Linux on IBM Z and LinuxONE, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0-ee upgrade-prepare
```
  - If you are upgrading to IBM Cloud Private fix pack version 3.2.0.2003, run the following command:
    - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0.2003-ee upgrade-prepare
```
    - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0.2003-ee upgrade-prepare
```
    - For Linux on IBM Z and LinuxONE, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0.2003-ee upgrade-prepare
```
  If the cluster preparation fails, review the error message and resolve any issues. Then, run the upgrade-prepare command again.
4. Upgrade Kubernetes:
  - If you are upgrading to IBM Cloud Private version 3.2.0, run the following command:
    - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0-ee upgrade-k8s
```
    - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0-ee upgrade-k8s
```
    - For Linux on IBM Z and LinuxONE, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0-ee upgrade-k8s
```
  - If you are upgrading to IBM Cloud Private fix pack version 3.2.0.2003, run the following command:
    - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0.2003-ee upgrade-k8s
```
    - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0.2003-ee upgrade-k8s
```
    - For Linux on IBM Z and LinuxONE, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0.2003-ee upgrade-k8s
```
  If the Kubernetes upgrade fails with a different message, review the error message and resolve any issues. Then, run the upgrade Kubernetes services command again.
5. Upgrade the charts:
  - If you are upgrading to IBM Cloud Private version 3.2.0, run the following command:
    - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0-ee upgrade-chart
```
    - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0-ee upgrade-chart
```
    - For Linux on IBM Z and LinuxONE, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0-ee upgrade-chart
```
  - If you are upgrading to IBM Cloud Private fix pack version 3.2.0.2003, run the following command:
    - For Linux x86_64, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-amd64:3.2.0.2003-ee upgrade-chart
```
    - For Linux on Power (ppc64le), run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-ppc64le:3.2.0.2003-ee upgrade-chart
```
    - For Linux on IBM Z and LinuxONE, run the following command:
```
sudo docker run -e LICENSE=accept --net=host --rm -t -v "$(pwd)":/installer/cluster \
ibmcom/icp-inception-s390x:3.2.0.2003-ee upgrade-chart
```
  If the chart upgrade fails with a different message, review the error message and resolve any issues. Then, re-run the upgrade chart command again.
6. If GlusterFS is installed in your cluster, you must upgrade the GlusterFS client to version 4.1.5.
7. Update NGINX ingress rewrite-target annotation. For more information, see NGINX ingress rewrite-target annotation fails when you upgrade to IBM Cloud Private Version 3.2.0.
Verify the status of your upgrade.
- If the upgrade succeeded, the access information for your cluster is displayed:
```
UI URL is https://<Cluster Master Host>:<Cluster Master API Port>
```
The <Cluster Master Host>:<Cluster Master API Port> value is defined in Master endpoint.
- If you encounter errors, see Troubleshooting.
Clear your browser cache.
If you customized the authentication URL before upgrade, you must add the customized URL to the registration.json file after you upgrade your cluster. For more information, see Customizing the authentication URL.
If you have either applications that use GPU resources or a resource quota for GPU resources, you need to manually update the application or resource quota with the new GPU resource name nvidia.com/gpu.
- For applications that use GPU resources, follow the steps in Creating a deployment with attached GPU resources to run a sample GPU application. For your own GPU application, you need to update the application to use the new GPU resource name nvidia.com/gpu. For example, to update the deployment properties, you can use either the management console (see Modifying a deployment) or the kubectl CLI.
- To update the resource quota for GPU resources, follow the steps in Setting resource quota to set a resource quota for your namespace. For upgrading, you need to update the resource quota to use the GPU resource name nvidia.com/gpu. For example, you can set the GPU quota to requests.nvidia.com/gpu: "2".
Access your cluster. From a web browser, browse to the URL for your cluster. For a list of supported browsers, see Supported browsers.
- For more information about accessing your cluster by using the IBM Cloud Private management console from a web browser, see Accessing your IBM Cloud Private cluster by using the management console.
- For more information about accessing your cluster by using the Kubernetes command line (kubectl), see Installing the Kubernetes CLI (kubectl).
  
  Note: After your upgrade the pod security policy for your clusters is automatically enabled, but set to the least restrictive setting to avoid access problems. See Pod security for information about how to manage the pod security policy settings.
- If you installed fix pack version 3.2.0.2003, add the root CA certificate to your trust store. With this fix pack, users on macOS 10.15 or newer cannot access the management console until the root CA certificate is added to the trust store. For more information, see:
  - Refreshing root CA certificates
  - Troubleshooting console access on macOS Catalina
Ensure that all the IBM Cloud Private default ports are open. For more information about the default IBM Cloud Private ports, see Default ports.
Back up the boot node. Copy your /<new_installation_directory>/cluster directory to a secure location.
If you use Cloud Automation Manager in your IBM Cloud Private cluster, you must also upgrade it. See Upgrading Cloud Automation Manager.
Clean up the obsolete charts. In the IBM Cloud Private 3.2.0 release, some charts, such as mariadb, unified-router, and heapster are removed; but are not automatically deleted in case you want to revert to the previous release. Only after you verify the upgrade and the cluster is operating well, run the following command to remove the obsolete charts:
```
helm delete --purge --tls --timeout=600  mariadb
helm delete --purge --tls --timeout=600  heapster
helm delete --purge --tls --timeout=600  unified-router
```
If you disabled single sign-on (SSO), re-enable it. For more information, see Configuring single sign-on.