Administering IBM Streams resources

Application resources are created by IBM® Streams when jobs are submitted. Management resources are created when the Streams instance and resources are created.

Resource allocation for application services

Application resources are dynamic in nature (added to or removed from an instance dynamically based upon the job or processing element requirements) and are not needed until actual jobs are submitted. A Streams scheduler component determines the application resources needed by the jobs and utilizes the Streams resource manager component residing in the management deployment to request and to release application resources as needed. A dynamic resource is released from the instance when the Streams instance is deleted or when the resource is no longer needed.

How job-scoped resources are allocated

  • When a job is submitted, Streams analyzes the application to determine the job's resource requirements.
  • You can use the updateOperators command to modify the CPU resources that are used by a running job by selecting a parallel region section of a running job and then increasing or decreasing the CPU resources that are used by the region. For more information, see streamtool updateoperators.

Although the Kubernetes resource manager ultimately controls how many resources (CPU cores and memory) are used by each individual resource, you can influence these values by supplying resource specifications to the Streams scheduler that are then passed to the Kubernetes resource manager.

The Kubernetes resource manager is given a resource specification that describes a range of CPU resources that are to be allocated to a pod, in terms of two values:

  • Request: The minimum number of CPU cores that are allocated to this resource
  • Limit: The maximum number of CPU cores that are allocated to this resource.

Based upon the current workload, the Kubernetes resource manager determines where within this range the actual allocation lies. Likewise, there are also specifications for the amount of memory to be allocated.

Although the Kubernetes resource manager decides the actual allocation, the Streams scheduler (with optional user specifications) determines how many resources to request, which operators to place on each resource, and the request and limit ranges to be used for each resource.
Default deployment
If no resource specification or placement constraints are specified with the job, the default deployment is to fuse all the operators into one processing element, and to place that processing element on one resource with the resource specification that is specified in the application template. (Each processing element is placed on its own resource.) However, if you require a custom deployment for the job, you have several mechanisms available to you depending on your requirements.
Number of resources
The number of resources that are requested is determined by the following factors:
  • Nothing specified. All operators are fused into one processing element, and placed on one resource.
  • Operator constraints. For example, the partitionIsolation parameter causes a separate processing element, and therefore a separate resource, to be needed.
  • Hostpool constraints. For example, members of different hostpools are not fused into a processing element on the same resource.
  • User-supplied resource numbers.
    • Job level constraints (that is, the number of resources for the total job)
    • Hostpool level constraints (that is, the number of resources for use by processing elements in this hostpool)
Resource specifications
The resource specifications for each resource are determined by the following methods. Each method is optional and is processed in the order listed.
  1. If no explicit user values are supplied, the default value (CPU and memory) that is defined in the application template under the corresponding tagging combination is used.
  2. Operator-level CPU specification = The request and limit CPU value to be used for the processing element that contains this operator.
  3. Operator-level memory specification = The request and limit memory value to be used for the processing element that contains this operator.
  4. Hostpool-level CPU specification = The total of the request and limit CPU values to be spread across all processing elements associated with this hostpool.
  5. Job-level CPU specification = The total of the request and limit CPU values to be spread across all processing elements in the entire job

The Streams scheduler evaluates these optionally defined specifications and spreads the resulting number of CPU cores proportionally according to the number of relevant processing elements.

Resource allocation for management services

Management resources are created and maintained by the Streams instance operator when a Streams instance is created. The instance operator creates Kubernetes StatefulSets for the various management resources, which tell Kubernetes to allocate and manage containers to run the management logic.