Changing required node settings

On premises or private cloud

1. On the load balancer node, check the HAProxy timeout settings in the /etc/haproxy/haproxy.cfg file.
   The recommended values are at least:

   timeout client          300s 
   timeout server          300s 

2. If the timeout values are less than 300 seconds (5 minutes), update the values:
   • To change the timeout client setting, enter the following command:

     sed -i -e "/timeout client/s/ [0-9].*/ 5m/" /etc/haproxy/haproxy.cfg

   • To change the timeout server setting, enter the following command:

     sed -i -e "/timeout server/s/ [0-9].*/ 5m/" /etc/haproxy/haproxy.cfg

3. Run the following command to apply the changes that you made to the HAProxy configuration:

   systemctl restart haproxy

On IBM Cloud

If you are setting HAProxy timeout settings for Cloud Pak for Data on IBM Cloud, you can configure route timeouts using the oc annotate command.

1. Use the following command to set the server-side timeout for the HAProxy route to 360 seconds:

   oc annotate route zen-cpd --overwrite haproxy.router.openshift.io/timeout=360s

   If you don't provide the units, ms is the default.

2. Optionally, customize other route-specific settings. For more information, see Route-specific annotations.

For Red Hat OpenShift version 4.5 or 4.6

On Red Hat OpenShift, you can use the Node Tuning Operator to manage node-level profiles. For more information, see Using the Node Tuning Operator.

1. Create a YAML file, 42-cp4d.yaml, with the following content. If your current settings are less than the recommendations, adjust the settings in your YAML file. This step assumes that you have worker nodes with 64 GB of RAM.

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: cp4d-wkc-ipc
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - name: cp4d-wkc-ipc
       data: |
         [main]
         summary=Tune IPC Kernel parameters on OpenShift Worker Nodes running WKC Pods
         [sysctl]
         kernel.shmall = 33554432
         kernel.shmmax = 68719476736
         kernel.shmmni = 32768
         kernel.sem = 250 1024000 100 32768
         kernel.msgmax = 65536
         kernel.msgmnb = 65536
         kernel.msgmni = 32768
         vm.max_map_count = 262144
     recommend:
     - match:
       - label: node-role.kubernetes.io/worker
       priority: 10
       profile: cp4d-wkc-ipc

   IBM Power Systems On Power Systems, create a YAML file, 42-cp4d.yaml, with the following content. Adjust kernel.shmall for a 64 K OS system page size for Power Systems.

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: cp4d-wkc-ipc
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - name: cp4d-wkc-ipc
       data: |
         [main]
         summary=Tune IPC Kernel parameters on OpenShift Worker Nodes running WKC Pods
         [sysctl]
         kernel.shmall = 2097152
         kernel.shmmax = 68719476736
         kernel.shmmni = 32768
         kernel.sem = 250 1024000 100 32768
         kernel.msgmax = 65536
         kernel.msgmnb = 65536
         kernel.msgmni = 32768
         vm.max_map_count = 262144
     recommend:
     - match:
       - label: node-role.kubernetes.io/worker
       priority: 10
       profile: cp4d-wkc-ipc

2. Run the following command to apply the changes:

   oc create -f 42-cp4d.yaml

For Red Hat OpenShift version 3.11

On each worker node in the cluster, perform the following steps.

1. Create a conf file for the Cloud Pak for Data kernel settings in the /etc/sysctl.d directory.

   The files are processed in alphabetical order. To ensure that the settings from the Cloud Pak for Data conf file are retained after the sysctl files are processed, give the conf file a name that ensures it is at the end of the queue. For example, name the file /etc/sysctl.d/42-cp4d.conf

2. Check the virtual memory limit setting by running the following command:

   sysctl -a 2>/dev/null | grep vm.max | grep -v next_id

   The recommended value is at least:

   vm.max_map_count = 262144

   1. If the vm.max_map_count value is less than 262144, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      vm.max_map_count = 262144

3. Check the message limit settings by running the following command:

   sysctl -a 2>/dev/null | grep kernel.msg | grep -v next_id

   The recommended values are at least:

   kernel.msgmax = 65536
   kernel.msgmnb = 65536
   kernel.msgmni = 32768

   1. If the kernel.msgmax value is less than 65536, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.msgmax = 65536

   2. If the kernel.msgmnb value is less than 65536, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.msgmnb = 65536

   3. If the kernel.msgmni value is less than 32768, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.msgmni = 32768

4. Check the shared memory limit settings by running the following command:

   sysctl -a 2>/dev/null | grep kernel.shm | grep -v next_id | grep -v shm_rmid_forced

   The recommended values are at least:

   kernel.shmmax = 68719476736
   kernel.shmall = 33554432
   kernel.shmmni = 32768

   1. If the kernel.shmmax value is less than 68719476736, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.shmmax = 68719476736

   2. If the kernel.shmall value is less than 33554432, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.shmall = 33554432

   3. If the kernel.shmmni value is less than 32768, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.shmmni = 32768

5. Check the semaphore limit settings by running the following command:

   sysctl -a 2>/dev/null | grep kernel.sem | grep -v next_id

   The recommended values are at least:

   kernel.sem = 250 1024000 100 32768

   Specifically:

   • The max semaphores per array must be at least 250.
   • The max semaphores system wide must be at least 1024000.
   • The max ops per semop call must be at least 100.
   • The max number of arrays must be at least 32768.
   1. If any of the semaphore limit settings are less that the minimum requirements, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.sem = 250 1024000 100 32768

6. Run the following command to apply the changes that you made to the /etc/sysctl.d/42-cp4d.conf:

   sysctl -p /etc/sysctl.d/42-cp4d.conf

Power SMT and slub_max_order settings

1. Label all small core KVM capable worker nodes that are not running Db2® Warehouse workloads to SMT=2. For example:

   oc label node <node> SMT=2 --overwrite

2. Label all small core KVM capable worker nodes that are running Db2 Warehouse workloads to SMT=4. For example:

   oc label node <node> SMT=4 --overwrite

3. Label all big core PowerVM capable worker nodes that are not running Db2 Warehouse workloads to SMT=4. For example:

   oc label node <node> SMT=4 --overwrite

4. Label all big core PowerVM capable worker nodes that are running Db2 Warehouse workloads to SMT=8. For example:

   oc label node <node> SMT=8 --overwrite 

5. Create a YAML file, smt.yaml, with the following content:
   Note: The ignition version value changes based on the Red Hat OpenShift version number.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 99-worker-smt
   spec:
     kernelArguments:
     - slub_max_order=0
     config:
       ignition:
         version: 2.2.0
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKZXhwb3J0IFBBVEg9L3Jvb3QvLmxvY2FsL2Jpbjovcm9vdC9iaW46L3NiaW46L2JpbjovdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluCmV4cG9ydCBLVUJFQ09ORklHPS92YXIvbGliL2t1YmVsZXQva3ViZWNvbmZpZwpDT1JFUFM9JCgvYmluL2xzY3B1IHwgL2Jpbi9hd2sgLUY6ICcgJDEgfiAvXkNvcmVcKHNcKSBwZXIgc29ja2V0JC8ge3ByaW50ICQyfSd8L2Jpbi94YXJncykKU09DS0VUUz0kKC9iaW4vbHNjcHUgfCAvYmluL2F3ayAtRjogJyAkMSB+IC9eU29ja2V0XChzXCkkLyB7cHJpbnQgJDJ9J3wvYmluL3hhcmdzKQpsZXQgVE9UQUxDT1JFUz0kQ09SRVBTKiRTT0NLRVRTCk1BWFRIUkVBRFM9JCgvYmluL2xzY3B1IHwgL2Jpbi9hd2sgLUY6ICcgJDEgfiAvXkNQVVwoc1wpJC8ge3ByaW50ICQyfSd8L2Jpbi94YXJncykKbGV0IE1BWFNNVD0kTUFYVEhSRUFEUy8kVE9UQUxDT1JFUwpDVVJSRU5UU01UPSQoL2Jpbi9sc2NwdSB8IC9iaW4vYXdrIC1GOiAnICQxIH4gL15UaHJlYWRcKHNcKSBwZXIgY29yZSQvIHtwcmludCAkMn0nfC9iaW4veGFyZ3MpCgpTTVRMQUJFTD0kKC9iaW4vb2MgZ2V0IG5vZGUgJEhPU1ROQU1FIC1MIFNNVCAtLW5vLWhlYWRlcnMgfC9iaW4vYXdrICd7cHJpbnQgJDZ9JykKaWYgW1sgLW4gJFNNVExBQkVMIF1dCiAgdGhlbgogICAgY2FzZSAkU01UTEFCRUwgaW4KICAgICAgMSkgVEFSR0VUU01UPTEKICAgIDs7CiAgICAgIDIpIFRBUkdFVFNNVD0yCiAgICA7OwogICAgICA0KSBUQVJHRVRTTVQ9NAogICAgOzsKICAgICAgOCkgVEFSR0VUU01UPTgKICAgIDs7CiAgICAgICopIFRBUkdFVFNNVD0kQ1VSUkVOVFNNVCA7IGVjaG8gIlNNVCB2YWx1ZSBtdXN0IGJlIDEsIDIsIDQsIG9yIDggYW5kIHNtYWxsZXIgdGhhbiBNYXhpbXVtIFNNVC4iCiAgICA7OwogICAgZXNhYwogIGVsc2UKICAgIFRBUkdFVFNNVD0kTUFYU01UCmZpCgp0b3VjaCAvcnVuL21hY2hpbmUtY29uZmlnLWRhZW1vbi1mb3JjZQoKQ1VSUkVOVFNNVD0kKC9iaW4vbHNjcHUgfCAvYmluL2F3ayAtRjogJyAkMSB+IC9eVGhyZWFkXChzXCkgcGVyIGNvcmUkLyB7cHJpbnQgJDJ9J3wvYmluL3hhcmdzKQoKaWYgW1sgJENVUlJFTlRTTVQgLW5lICRUQVJHRVRTTVQgXV0KICB0aGVuCiAgICBJTklUT05USFJFQUQ9MAogICAgSU5JVE9GRlRIUkVBRD0kVEFSR0VUU01UCiAgICBpZiBbWyAkTUFYU01UIC1nZSAkVEFSR0VUU01UIF1dCiAgICAgIHRoZW4KICAgICAgICB3aGlsZSBbWyAkSU5JVE9OVEhSRUFEIC1sdCAkTUFYVEhSRUFEUyBdXQogICAgICAgIGRvCiAgICAgICAgICBPTlRIUkVBRD0kSU5JVE9OVEhSRUFECiAgICAgICAgICBPRkZUSFJFQUQ9JElOSVRPRkZUSFJFQUQKCiAgICAgICAgICB3aGlsZSBbWyAkT05USFJFQUQgLWx0ICRPRkZUSFJFQUQgXV0KICAgICAgICAgIGRvCiAgICAgICAgICAgIC9iaW4vZWNobyAxID4gL3N5cy9kZXZpY2VzL3N5c3RlbS9jcHUvY3B1JE9OVEhSRUFEL29ubGluZQogICAgICAgICAgICBsZXQgT05USFJFQUQ9JE9OVEhSRUFEKzEKICAgICAgICAgIGRvbmUKICAgICAgICAgIGxldCBJTklUT05USFJFQUQ9JElOSVRPTlRIUkVBRCskTUFYU01UCiAgICAgICAgICB3aGlsZSBbWyAkT0ZGVEhSRUFEIC1sdCAkSU5JVE9OVEhSRUFEIF1dCiAgICAgICAgICBkbwogICAgICAgICAgICAvYmluL2VjaG8gMCA+IC9zeXMvZGV2aWNlcy9zeXN0ZW0vY3B1L2NwdSRPRkZUSFJFQUQvb25saW5lCiAgICAgICAgICAgIGxldCBPRkZUSFJFQUQ9JE9GRlRIUkVBRCsxCiAgICAgICAgICBkb25lCiAgICAgICAgICBsZXQgSU5JVE9GRlRIUkVBRD0kSU5JVE9GRlRIUkVBRCskTUFYU01UCiAgICAgICAgZG9uZQogICAgICBlbHNlCiAgICAgICAgZWNobyAiVGFyZ2V0IFNNVCBtdXN0IGJlIHNtYWxsZXIgb3IgZXF1YWwgdGhhbiBNYXhpbXVtIFNNVCBzdXBwb3J0ZWQiCiAgICBmaQpmaQo=
             verification: {}
           filesystem: root
           mode: 0755
           overwrite: true
           path: /usr/local/bin/powersmt
       systemd:
         units:
           - name: smt.service
             enabled: true
             contents: |
               [Unit]
               Description=Set SMT
               After=network-online.target
               Before= crio.service
               [Service]
               Type=oneshot
               RemainAfterExit=yes
               ExecStart=/usr/local/bin/powersmt
               [Install]
               WantedBy=multi-user.target
   

6. Run the following command to apply the changes:
   Note: You must ensure that the cluster master nodes (or control plane) are in Ready status before you issue this command.

   oc create -f smt.yaml

   Your worker nodes will perform a rolling reboot action to update the kernel argument slub_max_order and set the labeled SMT level.