Upgrading the IBM Cloud Private Docker package

Upgrade Docker engines that were installed by using the IBM Cloud Private Docker package.

Preparing your cluster for a patch
Upgrading the IBM Cloud Private Docker package
Upgrading the IBM Cloud Private Docker package (boot node)
Upgrading the IBM Cloud Private Docker package (cluster nodes)

Note: The following components are affected when you upgrade Docker:

All Docker containers are restarted. Some pods might not successfully start after you restart the Docker service. To resolve this issue, stop all related pods and let the pods restart.
The file /lib/systemd/system/docker.service is overwritten. To resolve this issue, reconfigure Docker after you upgrade. For more information, see Troubleshooting.
For more information on upgrading the Docker version to 19.03.15, see Upgrade to Docker 19.03.15 Fix Central

Preparing your cluster for a patch

Before you upgrade, ensure that your cluster meets the following requirements:

The /tmp directory on all nodes must have at least 2 GB of space and have executable permission granted.
For Ubuntu cluster nodes, the apt repository must be available.
For Red Hat Enterprise Linux (RHEL), the yum repository must be available.
Kubernetes CLI (kubectl) must be installed on your IBM Cloud Private boot node. For more information about installing kubectl, see Installing the Kubernetes CLI (kubectl).

Upgrading the IBM Cloud Private Docker package (boot node)

Upgrade a boot node that was installed by using the IBM Cloud Private Docker package.

Download the Docker package for your platform. To download Docker 19.03.15, see FixCentral.
If your boot node is one of your cluster nodes, see Upgrading the IBM Cloud Private Docker package (cluster nodes).
Upgrade Docker on your boot node.

Note: The following commands update Docker to version 18.09.7. Support newer versions of Docker is available when you apply the latest fix pack. The latest supported Docker version is 19.03.11, which is available when you apply the 3.2.1.2008 fix pack, or the 3.2.2.2008 fix pack, or a newer 3.2.1.x or 3.2.2.x fix pack. If you have applied one of these fix packs, replace the value 18.09.7 with 19.03.11 in the following commands to upgrade Docker.
- For Linux, run this command:
```
chmod +x icp-docker-18.09.7_x86_64.bin
sudo ./icp-docker-18.09.7_x86_64.bin --upgrade
```
- For Linux on Power (ppc64le), run this command:
```
chmod +x icp-docker-18.09.7_ppc64le.bin
sudo ./icp-docker-18.09.7_ppc64le.bin --upgrade
```
- For Linux® on IBM® Z and LinuxONE, run this command:
```
chmod +x icp-docker-18.09.7_s390x.bin
sudo ./icp-docker-18.09.7_s390x.bin --upgrade
```
  Note: To install Docker version 18.09.7 on RHEL 7.6, you must update the container-selinux to version 2.107 or later.
```
yum -y update container-selinux
rpm -qa | grep container-selinux
```
Ensure that Docker engine is started. Run the following command:
```
sudo systemctl start docker
```

Upgrading the IBM Cloud Private Docker package (cluster nodes)

Upgrade cluster nodes that were installed by using the IBM Cloud Private Docker package.

Switch to the /<installation_directory>/cluster/ directory.
```
 cd /<installation_directory>/cluster/
```
Edit the config.yaml file to set value for Docker_version to 18.09.7:
```
 docker_version: 18.09.7
```
Note: If you have applied the 3.2.1.2008 fix pack, or the 3.2.2.2008 fix pack, or a newer 3.2.1.x or 3.2.2.x fix pack, and are upgrading Docker to version 19.03.11, set the value to 19.03.11.
Back up and remove the PodDisruptionBudget resource.
1. Check whether PodDisruptionBudget is available in your cluster.
```
kubectl get PodDisruptionBudget --all-namespaces
```
2. If PodDisruptionBudget is available in your cluster, back up and remove PodDisruptionBudget before you upgrade Docker.
```
kubectl get PodDisruptionBudget --all-namespaces -oyaml > PodDisruptionBudget.yaml
kubectl delete -f PodDisruptionBudget.yaml
```
Upgrade Docker.
```
 sudo docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-$(uname -m | sed 's/x86_64/amd64/g'):3.2.1-ee upgrade-docker
```
You can also use the -l option to upgrade specific cluster nodes.
```
 sudo docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-$(uname -m | sed 's/x86_64/amd64/g'):3.2.1-ee upgrade-docker -l <node_IP_address>
```
- For worker nodes, use the -l worker.
- In HA environments, use the -l <host_ip> to upgrade master and proxy nodes one at a time.
  
  If your boot node is one of the cluster nodes, upgrade Docker on the boot node manually.
```
kubectl cordon <boot_node_IP>
kubectl drain <boot_node_IP> --ignore-daemonsets --delete-local-data --force --timeout=600s
./icp-docker-18.09.7_x86_64.bin --upgrade
kubectl uncordon <boot_node_IP>
```
  Notes:
  - For non-HA environments, if you fail to uncordon the node after you upgrade Docker, you need to configure kubectl to resolve the issue. For more information, see Troubleshooting.
  - If you have applied the 3.2.1.2008 fix pack, or the 3.2.2.2008 fix pack, or a newer 3.2.1.x or 3.2.2.x fix pack, and are upgrading Docker to version 19.03.11, replace 18.09.7 in the preceding command to 19.03.11.

Add PodDisruptionBudget back to your cluster.

Edit your PodDisruptionBudget.yaml backup file to remove the following status section:

status:
  currentHealthy: 1
  desiredHealthy: 2
  disruptedPods: null
  disruptionsAllowed: 0
  expectedPods: 1
  observedGeneration: 1

Re-create the PodDisruptionBudget resource.

kubectl apply -f PodDisruptionBudget.yaml

On all nodes, ensure that Docker engine is started by running the following command:
```
sudo systemctl start docker
```
Check the Docker version by running the following command:
```
kubectl get nodes -oyaml | grep containerRuntimeVersion
```
The Docker version must be 18.09.7.
```
containerRuntimeVersion: docker://18.09.7
```
Note: If you have applied the 3.2.1.2008 fix pack, or the 3.2.2.2008 fix pack, or a newer 3.2.1.x or 3.2.2.x fix pack, and are upgrading Docker to version 19.03.11, set the Docker version to 19.03.11.

You can also check the Docker version by running the following commands on your cluster nodes:
```
docker version
docker info
```
Check the pod status. All pods must be running after you upgrade Docker.
```
kubectl get pods --all-namespaces
```
The pod status must show as running.

Troubleshooting

If you encounter any issues during your upgrade, review the following common issues and errors that can occur for Docker upgrade.