Changing required node settings

For Red Hat® OpenShift® version 4.3

On Red Hat OpenShift version 4.3, Machine Config Pools manage your cluster of nodes and their corresponding Machine Configs (machineconfig). To change a setting in the crio.conf file, you can create a new machineconfig containing only the crio.conf file.

1. Make sure that you have python3 installed.
2. On any worker node in the cluster, extract the current crio.conf settings to /tmp/crio.conf.

   For example:

   python3 -c "import sys, urllib.parse; print(urllib.parse.unquote(sys.argv[1]))" $(oc get $(oc get mc -o name --sort-by=.metadata.creationTimestamp | grep rendered-worker| tail -n1 ) -o jsonpath='{.spec.config.storage.files[?(@.path=="/etc/crio/crio.conf")].contents.source}' | awk -F" " '{print $1}' | sed 's/data:,//g')  > /tmp/crio.conf

3. On the worker node, check the maximum number of open files setting by running the following command:

   ulimit -n

   The recommended value is at least 66560.

   1. If the ulimit value is less than 66560, edit or add the following entry in the [crio.runtime] section of the /tmp/crio.conf file:

      default_ulimits = [
              "nofile=66560:66560"
      ]

4. On the same worker node, check the maximum number of processes setting by running the following command:

   ulimit -u

   The recommended value is at least 12288.

   1. If the ulimit value is less than 12288, edit or add the following entry in the [crio.runtime] section of the /tmp/crio.conf file:

      pids_limit = 12288

5. Verify that /tmp/crio.conf looks like the following example:

   [crio]
   root = "/var/lib/containers/storage"
   runroot = "/var/run/containers/storage"
   storage_driver = "overlay"
   storage_option = [
           "overlay.override_kernel_check=1",
      ]
   
   [crio.api]
   listen = "/var/run/crio/crio.sock"
   stream_address = ""
   stream_port = "10010"
   file_locking = true
   
   [crio.runtime]
   runtime = "/usr/bin/runc"
   runtime_untrusted_workload = ""
   default_workload_trust = "trusted"
   no_pivot = false
   conmon = "/usr/libexec/crio/conmon"
   conmon_env = [
           "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      ]
   selinux = true
   seccomp_profile = "/etc/crio/seccomp.json"
   apparmor_profile = "crio-default"
   cgroup_manager = "systemd"
   hooks_dir_path = "/usr/share/containers/oci/hooks.d"
   default_mounts = [
           "/usr/share/rhel/secrets:/run/secrets",
      ]
   default_ulimits = [
           "nofile=66560:66560"
   ]
   pids_limit = 12288
   enable_shared_pid_namespace = false
   log_size_max = 52428800
   
   [crio.image]
   default_transport = "docker://"
   pause_image = "docker.io/openshift/origin-pod:v3.11"
   pause_command = "/usr/bin/pod"
   signature_policy = ""
   image_volumes = "mkdir"
   insecure_registries = [
   ""
   ]
   registries = [
   "docker.io"
   ]
   
   [crio.network]
   network_dir = "/etc/cni/net.d/"
   plugin_dir = "/opt/cni/bin"

6. Save the contents of the /tmp/crio.conf to a local variable by using the following command:

   crio_conf=$(cat /tmp/crio.conf | python3 -c "import sys, urllib.parse; print(urllib.parse.quote(''.join(sys.stdin.readlines())))")

   Important: Verify that crio.conf is not empty.
7. Create a new machineconfig, with the name 51-worker-cp4d-crio-conf, by using the following command:

   cat << EOF > /tmp/51-worker-cp4d-crio-conf.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
    labels:
      machineconfiguration.openshift.io/role: worker
    name: 51-worker-cp4d-crio-conf
   spec:
    config:
      ignition:
        version: 2.2.0
      storage:
        files:
        - contents:
            source: data:,${crio_conf}
          filesystem: root
          mode: 0644
          path: /etc/crio/crio.conf
   EOF

8. Apply the new machineconfig to the cluster by running the following command:

   oc create -f /tmp/51-worker-cp4d-crio-conf.yaml

For more information, see Changing Ignition Configs after installation.

For Red Hat OpenShift version 3.11

On each compute node in the cluster, perform the following steps.

1. Check the maximum number of open files setting by running the following command:

   ulimit -n

   The recommended value is at least 66560.

   1. If the ulimit value is less than 66560, edit or add the following entry in the [crio.runtime] section of the /etc/crio/crio.conf file:

      default_ulimits = [
              "nofile=66560:66560"
      ]

2. Check the maximum number of processes setting by running the following command:

   ulimit -u

   The recommended value is at least 12288.

   1. If the ulimit value is less than 12288, edit or add the following entry in the [crio.runtime] section of the /etc/crio/crio.conf file:

      pids_limit = 12288

3. Verify that /etc/crio/crio.conf is similar to the following example:

   [crio]
   root = "/var/lib/containers/storage"
   runroot = "/var/run/containers/storage"
   storage_driver = "overlay"
   storage_option = [
           "overlay.override_kernel_check=1",
      ]
   
   [crio.api]
   listen = "/var/run/crio/crio.sock"
   stream_address = ""
   stream_port = "10010"
   file_locking = true
   
   [crio.runtime]
   runtime = "/usr/bin/runc"
   runtime_untrusted_workload = ""
   default_workload_trust = "trusted"
   no_pivot = false
   conmon = "/usr/libexec/crio/conmon"
   conmon_env = [
           "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      ]
   selinux = true
   seccomp_profile = "/etc/crio/seccomp.json"
   apparmor_profile = "crio-default"
   cgroup_manager = "systemd"
   hooks_dir_path = "/usr/share/containers/oci/hooks.d"
   default_mounts = [
           "/usr/share/rhel/secrets:/run/secrets",
      ]
   default_ulimits = [
           "nofile=66560:66560"
   ]
   pids_limit = 12288
   enable_shared_pid_namespace = false
   log_size_max = 52428800
   
   [crio.image]
   default_transport = "docker://"
   pause_image = "docker.io/openshift/origin-pod:v3.11"
   pause_command = "/usr/bin/pod"
   signature_policy = ""
   image_volumes = "mkdir"
   insecure_registries = [
   ""
   ]
   registries = [
   "docker.io"
   ]
   
   [crio.network]
   network_dir = "/etc/cni/net.d/"
   plugin_dir = "/opt/cni/bin"

4. Run the following command to apply the changes that you made to the /etc/crio/crio.conf file:

   systemctl restart crio

For Red Hat OpenShift version 4.3

On Red Hat OpenShift, you can use the Node Tuning Operator to manage node-level profiles. For more information, see Using the Node Tuning Operator.

1. Create a YAML file, 42-cp4d.yaml, with the following content. If your current settings are less than the recommendations, adjust the settings in your YAML file. This step assumes that you have worker nodes with 64 GB of RAM.

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: cp4d-wkc-ipc
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - name: cp4d-wkc-ipc
       data: |
         [main]
         summary=Tune IPC Kernel parameters on OpenShift Worker Nodes running WKC Pods
         [sysctl]
         kernel.shmall = 33554432
         kernel.shmmax = 68719476736
         kernel.shmmni = 16384
         kernel.sem = 250 1024000 100 16384
         kernel.msgmax = 65536
         kernel.msgmnb = 65536
         kernel.msgmni = 32768
         vm.max_map_count = 262144
     recommend:
     - match:
       - label: node-role.kubernetes.io/worker
       priority: 10
       profile: cp4d-wkc-ipc

   IBM Power Systems On Power Systems, create a YAML file, 42-cp4d.yaml, with the following content. Adjust the kernel.sem if required.

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: cp4d-wkc-ipc
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - name: cp4d-wkc-ipc
       data: |
         [main]
         summary=Tune IPC Kernel parameters on OpenShift Worker Nodes running WKC Pods
         [sysctl]
         kernel.shmall = 2097152
         kernel.shmmax = 68719476736
         kernel.shmmni = 16384
         kernel.sem = 250 1024000 100 16384
         kernel.msgmax = 65536
         kernel.msgmnb = 65536
         kernel.msgmni = 32768
         vm.max_map_count = 262144
     recommend:
     - match:
       - label: node-role.kubernetes.io/worker
       priority: 10
       profile: cp4d-wkc-ipc

2. Run the following command to apply the changes:

   oc create -f 42-cp4d.yaml

For Red Hat OpenShift version 3.11

On each compute node in the cluster, perform the following steps.

1. Create a conf file for the Cloud Pak for Data kernel settings in the /etc/sysctl.d directory.

   The files are processed in alphabetical order. To ensure that the settings from the Cloud Pak for Data conf file are retained after the sysctl files are processed, give the conf file a name that ensures it is at the end of the queue. For example, name the file /etc/sysctl.d/42-cp4d.conf

2. Check the virtual memory limit setting by running the following command:

   sysctl -a 2>/dev/null | grep vm.max | grep -v next_id

   The recommended value is at least:

   vm.max_map_count = 262144

   1. If the vm.max_map_count value is less than 262144, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      vm.max_map_count = 262144

3. Check the message limit settings by running the following command:

   sysctl -a 2>/dev/null | grep kernel.msg | grep -v next_id

   The recommended values are at least:

   kernel.msgmax = 65536
   kernel.msgmnb = 65536
   kernel.msgmni = 32768

   1. If the kernel.msgmax value is less than 65536, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.msgmax = 65536

   2. If the kernel.msgmnb value is less than 65536, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.msgmnb = 65536

   3. If the kernel.msgmni value is less than 32768, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.msgmni = 32768

4. Check the shared memory limit settings by running the following command:

   sysctl -a 2>/dev/null | grep kernel.shm | grep -v next_id | grep -v shm_rmid_forced

   The recommended values are at least:

   kernel.shmmax = 68719476736
   kernel.shmall = 33554432
   kernel.shmmni = 16384

   1. If the kernel.shmmax value is less than 68719476736, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.shmmax = 68719476736

   2. If the kernel.shmall value is less than 33554432, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.shmall = 33554432

   3. If the kernel.shmmni value is less than 16384, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:
5. Check the semaphore limit settings by running the following command:

   sysctl -a 2>/dev/null | grep kernel.sem | grep -v next_id

   The recommended values are at least:

   kernel.sem = 250 1024000 100 16384

   Specifically:

   • The max semaphores per array must be at least 250.
   • The max semaphores system wide must be at least 1024000.
   • The max ops per semop call must be at least 100.
   • The max number of arrays must be at least 16384.
   1. If any of the semaphore limit settings are less that the minimum requirements, add the following entry in the /etc/sysctl.d/42-cp4d.conf file:

      kernel.sem = 250 1024000 100 16384

6. Run the following command to apply the changes that you made to the /etc/sysctl.d/42-cp4d.conf:

   sysctl -p /etc/sysctl.d/42-cp4d.conf