Configure the process limit for the nodes in the cluster

The number of processes that can be run inside a pod depends on the process limit that is configured in each worker node of the Red Hat® OpenShift® Container Platform cluster where the pods run.

The cluster administrator needs to configure the process limit for each of the worker nodes in the Red Hat OpenShift Container Platform cluster. The following instructions show how to change this limit to 4096 for an individual worker node. The instructions are for Red Hat OpenShift version 4.4 and later.

The method that you use to configure the process limit for your cluster depends on whether your cluster has the Machine Config Operator available or not. The following sections describe the different ways to configure the process limit. Use the section that applies to your cluster setup.

Changing the process limit by using Machine Config

Use the following instructions if the Machine Config Operator is available on your cluster:

Using the Red Hat OpenShift Container Platform command-line interface CLI, log in as a cluster administrator.

Run the following commands to create a ContainerRuntimeConfig custom resource.

cat <<EOF > custom-crio-configs.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: custom-crio-configs
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-crio: custom-crio-configs
  containerRuntimeConfig:
    pidsLimit: 4096
EOF

oc create -f custom-crio-configs.yaml

Run the following command to check that the custom resource was created.
```
oc get ContainerRuntimeConfig
```
The command output shows that the ContainerRuntimeConfig custom resource (CR) is created.
```
NAME                  AGE
custom-crio-configs   1h
```

Run the following command to edit the machineconfigpool custom resource file, which manages the worker nodes. Update the file to include the ContainerRuntimeConfig custom resource that you created.

oc edit machineconfigpool worker

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  creationTimestamp: "2020-12-17T00:07:51Z"
  generation: 11
  labels:
    custom-crio: custom-crio-configs <---- add this line to include the CR that was created in step 2
    machineconfiguration.openshift.io/mco-built-in: ""
    pools.operator.machineconfiguration.openshift.io/worker: ""

Run the following command to check that a new containerruntime object appears under the machineconfigs tag.

oc get machineconfigs | grep 'containerrun'

The command output is shown here.

99-worker-generated-containerruntime               721141df0005d407d15b6664982f22be0604eb72   3.1.0

Run the following command to monitor the configuration pool as the changes are rolled out to the worker nodes.
```
oc get mcp worker
```
The command output is shown here.
```
NAME    CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT
worker  rendered-worker-ee5  True     False     False     3
```
Note: All the worker nodes are restarted as part of the update roll-out. Therefore, the UPDATING column output shows False until all the nodes are restarted.
Verify that the pids_limit parameter was updated on the worker nodes by doing the following steps:
1. Run the following command to get the hostname of the worker node (worker hostname) that you want to verify.
```
oc get nodes
```
2. Run the following command on the worker node to initiate a debug session. Replace <worker hostname> with the actual hostname from step 7.a.
```
oc debug node/<worker hostname>
```
3. Run the following command to change the root directory to the root directory of the host.
```
chroot /host
```
4. Run the following command to verify that the pids_limit parameter was changed in the configuration.
```
crio config | grep 'pids_limit'
```

Changing the process limit without using Machine Config

Use the following instructions if the Machine Config Operator is not available on your cluster:

Apply the following YAML to your Red Hat OpenShift Container Platform cluster:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: pid-limit-daemonset
spec:
  selector:
    matchLabels:
      name: pid-limit-daemonset
  template:
    metadata:
      labels:
        name: pid-limit-daemonset
    spec:
      containers:
      - name: pid-limit-container
        image: us.icr.io/armada-master/alpine:latest
        command:
          - "/bin/sh"
          - "-c"
          - >
            sysctl -w kernel.pid_max=4096 &&
            while true; do sleep 10000; done
        securityContext:
          privileged: true
      tolerations:
      - key: node-role.kubernetes.io/worker
        operator: Exists
        effect: NoSchedule

Verify that the pids_limit parameter was updated on the worker nodes by doing the following steps:
1. Run the following command to get the hostname of the worker node (worker hostname) that you want to verify.
```
oc get nodes
```
2. Run the following command on the worker node to initiate a debug session. Replace <worker hostname> with the actual hostname from step 2.a.
```
oc debug node/<worker hostname>
```
3. Run the following command to change the root directory to the root directory of the host.
```
chroot /host
```
4. Run the following command to verify that the pids_limit parameter was changed in the configuration.
```
crio config | grep 'pids_limit'
```