IBM Support

Sweet32 and Weak Ciphers

Question & Answer


Question

We have been reported for a security vulnerability in our ICP cluster, related to weak ciphers currently in use (SWEET32).
How can this be manually fixed?

Answer

Following instructions have been successfully tested for an ICP 3.1.2 environment.
The changes must be implemented on ETCD, KUBERNETES CONTROL PLANE and KUBELET nodes.
- ETCD and KUBERNETES CONTROL PLANE: on all master nodes only
- KUBELET: on master nodes and also all cluster nodes

ETCD

1. Copy and back up etcd static pod manifest file on master node.

cp /etc/cfc/pods/etcd.json ~/icp-backup/
cp ~/icp-backup/etcd.json ~/icp-backup/etcd.json.bak

2. Update etcd static pod manifest file vim ~/icp-backup/etcd.json to add following options

"--cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"

By default etcd is using TLSv1.2. No need to update TLS min version.

3. Copy the updated etcd static pod manifest back. Kubelet will help restart etcd service.

cp ~/icp-backup/etcd.json /etc/cfc/pods/

4. Verify etcd service is started.

# docker ps |grep etcd
416e7e7ed2a5    33bdcac177c2    "etcd --name=etcd0 -…"  2 minutes ago    Up 2 minutes     k8s_etcd_k8s-etcd-9.21.55.15_kube-system_ae53b0c24e347e2f786003f83ab595b7_0

Run above steps on all master nodes.

KUBELET

1. Update kubelet service config vim /etc/cfc/kubelet/kubelet-service-config and add following option

tlsCipherSuites: ["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256"]

By default TLSv1.2 is used. No need to specify tls min version.

2. Restart kubelet service

systemctl restart kubelet
systemctl status kubelet

3. Verify kubelet is started.
  Check kubelet log

journalctl -u kubelet.service -f

Check node status

kubectl get nodes

Run above steps on all cluster nodes.

KUBERNETES CONTROL PLANE

1. Copy and back up Kubernetes static pod manifest file on master node.

cp /etc/cfc/pods/master.json ~/icp-backup/
cp ~/icp-backup/master.json ~/icp-backup/master.json.bak

2. Update Kubernetes static pod manifest file vim ~/icp-backup/master.json to add following options. There are 3 containers in the static pod, kube-controller-manager, kube-apiserver and kube-scheduler. Add same option for all 3 container.

( NOTE: "tls-cipher-suites" can be ignored for "kube-scheduler" container )

"--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"

3. Copy the updated Kubernetes static pod manifest back. Kubelet will help restart Kubernetes service.

cp ~/icp-backup/master.json /etc/cfc/pods/

4. Verify Kubernetes service is started.

# docker ps |grep hyperkube
b4844586cc1a    a28dcbcae557     "/hyperkube schedule…"  14 minutes ago   Up 14 minutes              k8s_scheduler_k8s-master-9.21.55.15_kube-system_40af6d00537d84138a6f8acab99c123a_3
4472e7a9f4bd    a28dcbcae557     "/hyperkube controll…"  14 minutes ago   Up 14 minutes              k8s_controller-manager_k8s-master-9.21.55.15_kube-system_40af6d00537d84138a6f8acab99c123a_3
bbc2b79cee2e    a28dcbcae55 7    "/hyperkube apiserve…"  20 hours ago    Up 14 minutes               k8s_apiserver_k8s-master-9.21.55.15_kube-system_40af6d00537d84138a6f8acab99c123a_0

Verify kubernetes service

kubectl get pods --all-namespaces

Run above steps on all master nodes.

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"3.1.2","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Product Synonym

ICP

Document Information

Modified date:
15 October 2019

UID

ibm11087504