Placing pods on particular nodes

Taints plus tolerations, plus node affinity provides complete pod placement control.

Taints and tolerations tell nodes what to reject, and protect nodes from unwanted workloads. However, they do not guide pods to specific nodes. Node affinity provides positive selection. It tells pods where to go based on node labels. However, it does not protect nodes from other pods.

Together taints, tolerations, and node affinity give you complete control. Combined, taints, tolerations, and node affinity provide fine-grained control over workload placement.

Taints protect the dedicated nodes.
Tolerations grant access to protected nodes.
Node affinity can select the correct nodes.

The following sections provide more information:

Node taints

Taints tell the node what it rejects.

Taints are applied to nodes to repel pods that do not have matching tolerations. A node that is tainted with NoSchedule for example rejects all pods except pods that tolerate the taint.

The following table shows the effect of the taint on a node.

Effect	Behavior	Use case
`NoSchedule`	Hard restriction Pods without toleration are NOT scheduled.	Strict workload isolation
`PreferNoSchedule`	Soft restriction Scheduler tries to avoid but is not guaranteed.	Preferred placement
`NoExecute`	Runtime enforcement Evicts existing pods that do not tolerate the taint.	Dynamic workload migration

The following example shows the syntax to create a taint.

oc taint nodes <node-name> <key>=<value>:<effect>

The following example shows how to dedicate a node to Content Platform Engine (CPE) workloads.

oc taint nodes cpe-node-1 component=cpe:NoSchedule

The following example shows how not to schedule on a maintenance node.

oc taint nodes maint-node-1 maintenance=true:PreferNoSchedule

The following example shows how to evict pods that do not tolerate the taint from a node that is being decommissioned.

oc taint nodes old-node-1 decommission=true:NoExecute

Pod tolerations

Tolerations tell the pod what restrictions that it can tolerate.

Tolerations are specified in the pod specification to allow the pod to be scheduled on tainted nodes. Each CP4BA component can independently define its own tolerations. Pods can have multiple tolerations to handle complex taint scenarios.

Note: If a node has multiple taints, a pod must have a matching toleration for each taint to be scheduled on that node.

The following configuration shows how to set toleration for the Content Platform Engine (CPE) component.

apiVersion: icp4a.ibm.com/v1
kind: ICP4ACluster
metadata:
  name: icp4adeploy
  namespace: cp4ba
spec:
  ecm_configuration:
    cpe:
      replica_count: 2
      tolerations:
        - key: "component"
          operator: "Equal"
          value: "cpe"
          effect: "NoSchedule"

The following table shows the operators for pod tolerations.

Operator	Behavior	Example
`Equal`	The key and value must match exactly.	`key: "component"`, `value: "cpe"`
`Exists`	A key must exist.	`key: "component"` (any value)

The following example shows how to create a toleration that needs an exact match.

tolerations:
  - key: "component"
    operator: "Equal"
    value: "cpe"
    effect: "NoSchedule"

The following example shows how to create a toleration that accepts any value.

tolerations:
  - key: "dedicated"
    operator: "Exists"
    effect: "NoSchedule"

The following example shows how to create a toleration that accepts all taints on a node.

tolerations:
  - operator: "Exists"

The following example shows how to create a toleration that accepts a specific effect only.

tolerations:
  - key: "maintenance"
    operator: "Equal"
    value: "true"
    effect: "PreferNoSchedule"

The following example shows that you can omit the tolerations parameter entirely or set it as an empty value.

# Undefined (uses Kubernetes defaults)
ecm_configuration:
  cpe:
    replica_count: 2
    # tolerations not specified

# Explicitly empty (also uses Kubernetes defaults)
ecm_configuration:
  cpe:
    replica_count: 2
    tolerations: []

Node affinity

The Kubernetes scheduler is used to place a pod on particular nodes. You can use node affinity to declare a pod to specific nodes with particular labels. Node affinity is used to specify where to deploy the Cloud Pak for Business Automation pods.

The following example shows the pods for the "cpe" component are put on worker nodes that are located in the "topology.kubernetes.io/zone: us-west-1a" and "topology.kubernetes.io/zone: us-west-1b" zones.

ecm_configuration:
  cpe:
    node_affinity:
      custom_node_selector_match_expression:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-west-1a
          - us-west-1b

Using custom labels and annotations to specify where to deploy a pod is versatile. The following example shows the worker node selection for the "cpe" pods is done by region. The node selection, for example, might help track resource usage by pods in us-west-1 and us-west-2.

ecm_configuration:
  cpe:
    node_affinity:
      custom_node_selector_match_expression:
       - key: topology.kubernetes.io/region
         operator: In
         values:
         - us-west-1
         - us-west-2

The operator parameter represents the relationship between the label on the node and the set of values in the custom_node_selector_match_expression parameters. The value of the operator parameter can be In or NotIn, Exists or DoesNotExist, Lt or Gt.

The value of the key parameter (label) must be matched to apply the operator value (rule).

Note: The node_affinity parameter can be set at the component level (ecm_configuration.cpe.node_affinity) and at the capability level (ecm_configuration.node_affinity). If the values are different, then the component-level value takes precedence over the value set at the capability level.