Node Licensing

Node Licensing explained

Instead of licensing based on the spec.template.spec.containers[].resources.limits.cpu values of the chargeable workload, the workload is licensed based on the CPU capacity of the worker node.

When the License Service analyzes the data it has gathered, it checks the value of the CPU limits that are allocated to pods on each worker node. If the spec.template.spec.containers[].resources.limits.cpu total that is assigned to pods on a worker exceeds that worker's actual size (number of cores), the License Service only assigns VPU licenses up to the size of that worker.

For example, suppose you have 100 pods running on worker 1, each with the following CPU resource allocations:

• spec.template.spec.containers[].resources.requests.cpu = 100m (0.1 core)
• spec.template.spec.containers[].resources.limits.cpu = 1000m (1 core)

If worker 1 has 24 cores, instead of paying for 100 cores worth of VPU licenses, you would pay only for 24 cores. This licensing model is called Node Licensing.

Node Licensing is best suited to workloads that involve high initialization startup CPU usage or that have very flexible workloads where the flows are not all being exercised all the time at a constant rate.

Worked example

Company ACME uses App Connect Enterprise in Kubernetes with the following configuration.

Cluster details

• Number of worker nodes: 36
• Each worker specs: 24 CPU cores/48 GB RAM
• Total cluster capacity: 864 CPU cores (36 x 24)

Workload characteristics

• Number of pods: approximately 700
• Workload behavior:
  • CPU usage is variable throughout the day.
  • Initial container startup requires a high CPU limit (approximately 500m).
  • After initialization, steady-state CPU usage is low (approximately 100m).

Current licensing situation

• Default resource limits cause workloads to be spread across many nodes.
• Total licensed cores based on limits is approximately 700 x 0.5 = 350 cores.
• This leads to higher licensing costs.

Proposed solution: Move to node-based licensing

• Instead of licensing based on total allocated cores, license per node.
• Potential benefits:
  • Simplifies cost calculation.
  • Might reduce overall license cost by leveraging the full node capacity rather than the sum of pod limits.
  • Pods are free to have virtually unlimited CPU limits to use if needed, which allows for better workload performance and startup speeds.

First, we must work out the optimal number of workers while still allowing n number of worker failures. In this scenario, we are going to presume that only 1 node at any given time would be removed (that is, for maintenance in a rolling update sequence).

7 worker configuration

If we distribute all 700 runtimes evenly across 7 workers, each worker will run approximately 100 containers. Under normal operations, each container uses about 0.1 CPU cores.

• Total CPU usage per worker:

  100 containers x 0.1 cores = 10 cores

• Available CPU cores remaining per worker:

  Assuming each node has around 24 cores, this leaves approximately:

  24 - 10 = 14 cores free on each node

Using this configuration, the License Service will count only 7 × 24 = 168 cores for licensing purposes. This is nearly half the number of cores compared to distributing workloads across the entire cluster, which uses 350 cores.

Worker failure scenario

• If a worker fails, 100 App Connect Enterprise instances (each requiring 0.5 cores to start) will be rescheduled across the remaining workers.
• To start all 100 instances quickly at the same time, a total of 50 cores are needed (100 × 0.5 cores).
• These 50 cores will be distributed evenly among the remaining 6 workers:
  • Each worker will handle 12.5 cores, equivalent to 25 instances.
• Each worker has 14 free cores by default, so they can accommodate the additional 12.5 cores with some headroom.
  • This headroom accounts for slight variations in resource requirements for a small percentage of containers.
  • This headroom allows for all pods to start at the same time. However, it’s not required and less headroom would just delay the startup of all flows by a few minutes.
• After startup, each node will run approximately 125 instances (original and rescheduled) using:

  125 x 0.1 (cores) = 12.5 (cores)

Additional considerations for moving to Node Licensing

• Cluster autoscaling
  • If autoscaling is enabled, ensure that it aligns with node-based licensing constraints.
• Monitoring and optimization
  • Continuously monitor actual versus requested CPU usage.
  • Adjust workload scheduling policies if needed to optimize node utilization.
• Cost analysis
  • Compare current core-based license fees against projected node-license fees.
  • Factor in potential savings from reduced number of licensed cores versus fixed per-node costs.
• Complete restart
  • The initial deployment must be staged; that is, when deploying the flows, they must not all be deployed at the same time since there is not enough capacity across all the workers for every flow to start. If this does occur, the containers would still start, but might take longer due to a reduced amount of CPU being made available to them.
• Number of failures
  • If more than 1 worker fails, it’s possible to recover, but startup times might be affected. If you want to fully tolerate more failures, you could add some extra workers such that you have enough headroom across the remaining workers to host the workload from the failed nodes. Be aware that if you try tolerate 3/4 worker failures, you might end up needing so many workers that Node Licensing provides no benefit over normal workload placement.

Additional benefits

This configuration also allows you to use even larger limits if desired. Provided you keep your cluster sized appropriately, you could set limits of 1 core on each container. So, during normal running where containers are being created individually, they would actually start up even quicker than the required limits and only in the failure scenario would it be closer to the maximum allowed time. 

Scheduling App Connect Enterprise workloads on specific worker nodes

To ensure that all your App Connect Enterprise workloads run only on a specific subset of worker nodes, use a combination of nodeSelectors, taints, and tolerations.

Step 1: Label and taint your worker nodes

First, identify the nodes that you want to dedicate to App Connect Enterprise workloads. For each node, complete these steps:

• Add a label to enable integration runtimes to be scheduled onto that node.
• Add a taint to prevent other workloads from being scheduled on that node.

You can choose any label and taint keys/values, but in this example we'll use:

• Label: workloadType=ace
• Taint: workloadType=ace:NoSchedule

Apply the label by using this command, where nodeName is your actual node name:

kubectl label nodes nodeName workloadType=ace

Then, add the taint:

kubectl taint nodes nodeName workloadType=ace:NoSchedule

Step 2: Update nodeSelector and tolerations in the IntegrationRuntime resource

For each IntegrationRuntime resource, update the pod spec settings in the CR to include the matching nodeSelector and corresponding toleration so that the integration runtime can be scheduled on these tainted nodes:

spec:
  template:
    spec:
      nodeSelector:
        workloadType: ace
      tolerations:
        - key: workloadType
          operator: Equal
          value: ace
          effect: NoSchedule

Result

After applying these changes, the following result is expected:

• The integration runtime pods will be scheduled only on the designated App Connect Enterprise worker nodes.
• Other pods (except essential Kubernetes system pods like DNS or ingress) will not run on these dedicated workers.

References

• Taints and Tolerations Kubernetes reference
• nodeSelector Kubernetes reference
• License Service for IBM Cloud Paks and IBM standalone containerized software
• Improved scheduling of integrations blog