Changing kernel parameter settings

To change the kernel parameter settings:

1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:

   ${CPDM_OC_LOGIN}

2. Check whether there is an existing kubeletconfig on the cluster:

   oc get kubeletconfig

3. Take the appropriate action based on whether the command returns the name of a kubeletconfig:
   • If the preceding command returns an existing kubeletconfig:
     1. Set the KUBELET_CONFIG environment variable to the name of the existing kubeletconfig:

        export KUBELET_CONFIG=<kubeletconfig-name>

     2. Run the following command to patch the kubeletconfig:

        oc patch kubeletconfig ${KUBELET_CONFIG} \
        --type=merge \
        --patch='{"spec":{"kubeletConfig":{"allowedUnsafeSysctls":["kernel.msg*", "kernel.shm*", "kernel.sem"]}}}'

   • If the preceding command returns No resources found, run the apply-db2-kubelet command:

     cpd-cli manage apply-db2-kubelet

Before you install the services on your cluster, ensure that you complete Specifying the privileges that Db2U runs with.

Decide whether you plan to deploy the services on dedicated nodes. With dedicated nodes, you can limit node tuning to the nodes where the service or services will run.

For more information about setting up dedicated nodes, see Setting up dedicated nodes for your Db2 Warehouse deployment.

The Node Tuning Operator helps you manage node-level tuning by orchestrating the tuned daemon. Tuned is a system tuning service for Linux®. The core of Tuned are profiles, which tune your system for different use cases. In addition to static application of system settings, Tuned can also monitor your system and optimize the performance on-demand based on the profile that is applied.

Tuned is distributed with a number of predefined profiles. However, it is also possible to modify the rules defined for each profile and customize how and what to tune. Tuned supports various types of system configuration such as sysctl, sysfs, and kernel boot parameters. For more information, see Monitoring and managing system status and performance and The Tuned Project

The Node Tuning Operator provides a unified management interface to users of node-level sysctls and gives more flexibility to add custom tuning.

The operator manages the containerized tuned daemon for Red Hat OpenShift Container Platform as a Kubernetes DaemonSet. It ensures the custom tuning specification is passed to all containerized tuned daemons that run in the cluster in the format that the daemons understand. The daemons run on all nodes in the cluster, one per node.

The Node Tuning Operator is part of a standard Red Hat OpenShift Container Platform installation. For more information, see Using the Node Tuning Operator in the Red Hat OpenShift documentation:

• Version 4.12
• Version 4.14
• Version 4.15
• Version 4.16

You can employ the Node Tuning Operator by using one of the following methods:

Creating a custom resource file

The custom resource method requires you to manually compute all required IPC kernel parameters.

1. Create a Tuned custom resource file.

   Use the following guidance to adjust the contents of the custom resource file:

   • You must compute the values for the IPC kernel parameters that are denoted by < ...>. Use the formulas in Kernel parameter requirements (Linux).
   • Use the memory.resource limit that you plan to apply to the deployment as size of RAM if your Kubernetes worker node pool is heterogeneous.
   • The match label icp4data and the corresponding value is only required for dedicated deployments.
     • If you use dedicated nodes, the IPC kernel tuning is applied only on the labeled worker nodes.
     • If you don't use dedicated nodes, remove the following lines from the custom resource:

           - label: icp4data
             value: database-db2wh

   • The custom resource injects the IPC sysctl changes on top of the default tuned profile settings on the OpenShift worker nodes.

   The following sample YAML file describes the basic structure that is needed to create the custom resource for a Node Tuning Operator instance that can tune IPC kernel parameters.

   apiVersion: tuned.openshift.io/v1
   kind: Tuned
   metadata:
     name: db2u-ipc-tune
     namespace: openshift-cluster-node-tuning-operator
   spec:
     profile:
     - name: openshift-db2u-ipc
       data: |
         [main]
         summary=Tune IPC Kernel parameters on OpenShift nodes running Db2U engine PODs
         include=openshift-node
   
         [sysctl]
         kernel.shmmni = <shmmni>
         kernel.shmmax = <shmmax>
         kernel.shmall = <shmall>
         kernel.sem = <SEMMSL> <SEMMNS> <SEMOPM> <SEMMNI>
         kernel.msgmni = <msgmni>
         kernel.msgmax = <msgmax>
         kernel.msgmnb = <msgmnb>
   
     recommend:
     - match:
       - label: node-role.kubernetes.io/worker
       - label: icp4data
         value: database-db2wh
       priority: 10
       profile: openshift-db2u-ipc

2. Save the custom resource as a YAML file. For example: /tmp/Db2UnodeTuningCR.yaml.
3. Log in to the cluster as a cluster administrator and create the custom resource:

   oc create -f /tmp/Db2UnodeTuningCR.yaml

It might take a few minutes for the custom resource to be created and for the custom IPC tuned profile to be applied on the worker nodes.

Creating a shell script

The shell script enables you to generate a YAML file that you can install, deploy, and run on the target OpenShift cluster.

The shell script automatically calculates the required IPC kernel parameters for you.

The following sample shell script can be used to do the following:

• Generate a YAML file that you can install and deploy and run on the target OpenShift cluster.
• Delete the custom resource and clean up deployed tuned profiles.

The sample assumes that the script is saved as /root/script/crtNodeTuneCR.sh.

#!/bin/bash

# Compute IPC kernel parameters as per IBM Documentation topic
# https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.qb.server.doc/doc/c0057140.html
# and generate the Node Tuning Operator CR yaml.

tuned_cr_yaml="/tmp/Db2UnodeTuningCR.yaml"
mem_limit_Gi=0
node_label=""
cr_name="db2u-ipc-tune"
cr_profile_name="openshift-db2u-ipc"
cr_namespace="openshift-cluster-node-tuning-operator"
create_cr="false"
delete_cr="false"

usage() {
    cat <<-USAGE #| fmt
    Usage: $0 [OPTIONS] [arg]

    OPTIONS:
    =======
    * -m|--mem-limit mem_limit  : The memory.limit (Gi) to be applied to Db2U deployment.
    * [-l|--label node_label]   : The node label to use for dedicated Cp4D deployments.
    * [-f|--file yaml_output]   : The NodeTuningOperator CR YAML output file. Default /tmp/Db2UnodeTuningCR.yaml.
    * [-c|--create]             : Create the NodeTuningOperator CR ${cr_name} using the generated CR yaml file.
    * [-d|--delete]             : Delete the NodeTuningOperator CR ${cr_name}.
    * [-h|--help]               : Display the help text of the script.
USAGE
}

[[ $# -lt 1 ]] && { usage && exit 1; }

while [[ $# -gt 0 ]]; do
    case "$1" in
        -f|--file) shift; tuned_cr_yaml=$1
        ;;
        -m|--mem-limit) shift; mem_limit_Gi=$1
        ;;
        -l|--label) shift; node_label=$1
        ;;
        -c|--create) create_cr="true"
        ;;
        -d|--delete) delete_cr="true"
        ;;
        -h|--help) usage && exit 0
        ;;
        *) usage && exit 1
        ;;
	esac
	shift
done

((ram_in_BYTES=mem_limit_Gi * 1073741824))
((ram_GB=ram_in_BYTES / (1024 * 1024 * 1024)))
((IPCMNI_LIMIT=32 * 1024))
tr ' ' '\n' < /proc/cmdline | grep -q ipcmni_extend && ((IPCMNI_LIMIT=8 * 1024 * 1024))

#
### =============== functions ================ ###
#
# Compute the required kernel IPC parameter values
compute_kernel_ipc_params() {
    local PAGESZ=$(getconf PAGESIZE)

    # Global vars
    ((shmmni=256 * ram_GB))
    shmmax=${ram_in_BYTES}
    ((shmall=2 * (ram_in_BYTES / PAGESZ)))
    ((msgmni=1024 * ram_GB))
    msgmax=65536
    msgmnb=${msgmax}
    SEMMSL=250
    SEMMNS=256000
    SEMOPM=32
    SEMMNI=${shmmni}

    # RH bugzilla https://access.redhat.com/solutions/4968021. Limit SEMMNI, shmmni and msgmni to the max
    # supported by the Linux kernel -- 32k (default) or 8M if kernel boot parameter 'ipcmni_extend' is set.
    ((SEMMNI=SEMMNI < IPCMNI_LIMIT ? SEMMNI : IPCMNI_LIMIT))
    ((shmmni=shmmni < IPCMNI_LIMIT ? shmmni : IPCMNI_LIMIT))
    ((msgmni=msgmni < IPCMNI_LIMIT ? msgmni : IPCMNI_LIMIT))
}

# Generate NodeTuning Operator YAML file
gen_tuned_cr_yaml() {
    # Generate YAML file for NodeTuning CR and save as ${tuned_cr_yaml}
    cat <<-EOF > ${tuned_cr_yaml}
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: ${cr_name}
  namespace: ${cr_namespace}
spec:
  profile:
  - name: ${cr_profile_name}
    data: |
      [main]
      summary=Tune IPC Kernel parameters on OpenShift nodes running Db2U engine PODs
      include=openshift-node

      [sysctl]
      kernel.shmmni = ${shmmni}
      kernel.shmmax = ${shmmax}
      kernel.shmall = ${shmall}
      kernel.sem = ${SEMMSL} ${SEMMNS} ${SEMOPM} ${SEMMNI}
      kernel.msgmni = ${msgmni}
      kernel.msgmax = ${msgmax}
      kernel.msgmnb = ${msgmnb}

  recommend:
  - match:
    - label: node-role.kubernetes.io/worker
EOF

    # Add the optional dedicated label into match array
    if [[ -n "${node_label}" ]]; then
        cat <<-EOF >> ${tuned_cr_yaml}
    - label: icp4data
      value: ${node_label}
EOF
    fi

    # Add the priority and profile keys
    cat <<-EOF >> ${tuned_cr_yaml}
    priority: 10
    profile: ${cr_profile_name}
EOF

    [[ "${create_cr}" == "true" ]] && return
    cat <<-MSG
===============================================================================
* Successfully generated the Node Tuning Operator Custom Resource Definition as
  ${tuned_cr_yaml} YAML with Db2U specific IPC sysctl settings.

* Please run 'oc create -f ${tuned_cr_yaml}' on the master node to
  create the Node Tuning Operator CR to apply those customized sysctl values.
===============================================================================
MSG
}

create_tuned_cr() {
    echo "Creating the Node Tuning Operator Custom Resource for Db2U IPC kernel parameter tuning ..."
    oc create -f ${tuned_cr_yaml}
    sleep 2

    # List the NodeTuning CR and describe
    oc -n ${cr_namespace} get Tuned/${cr_name}
    echo ""

    echo "The CR of the Node Tuning Operator deployed"
    echo "--------------------------------------------"
    oc -n ${cr_namespace} describe Tuned/${cr_name}
    echo ""
}

delete_tuned_cr() {
    echo "Deleting the Node Tuning Operator Custom Resource used for Db2U IPC kernel parameter tuning ..."
    oc -n ${cr_namespace} get Tuned/${cr_name} --no-headers -ojsonpath='{.kind}' | grep -iq tuned || \
        { echo "No matching CR found ..." && exit 0; }
    oc -n ${cr_namespace} delete Tuned/${cr_name}
    echo ""
    sleep 2

    # Get the list of containerized tuned PODs (DaemonSet) deployed on the cluster
    local tuned_pods=( $(oc -n ${cr_namespace} get po --selector openshift-app=tuned --no-headers -ojsonpath='{.items[*].metadata.name}') )
    # Remove the tuned profile directory deployed on those PODs
    for p in "${tuned_pods[@]}"; do
        echo "Removing the installed tuned profile ${cr_profile_name} on POD: $p"
        oc -n ${cr_namespace} exec -it $p -- bash -c "rm -fr /etc/tuned/${cr_profile_name}"
    done
    echo ""
}

#
### ================== Main ==================== ###
#

[[ "${delete_cr}" == "true" ]] && { delete_tuned_cr && exit 0; }

compute_kernel_ipc_params

gen_tuned_cr_yaml

[[ "${create_cr}" == "true" ]] && create_tuned_cr

Before you install the services on your cluster, ensure that you complete Specifying the privileges that Db2U runs with.