Kubernetes Deployments

You can create a Control Hub Kubernetes deployment for an active Kubernetes environment.

When you create the deployment, you define the engine type, version, and configuration to deploy to the Kubernetes cluster. Each engine instance runs in a dedicated Kubernetes pod. You also configure details about the pods, including whether the number of pods are automatically scaled during times of peak performance or whether a fixed number of pods are created.

If you are an advanced Kubernetes user, you can use advanced mode to directly edit the deployment YAML file.

When you start a Control Hub Kubernetes deployment, the IBM StreamSets Kubernetes agent that corresponds to the parent environment creates a YAML file describing the required resources. The YAML file creates a single Kubernetes deployment and secret in the Kubernetes namespace. The YAML file also creates a horizontal pod autoscaler if the Control Hub deployment is configured to allow legacy autoscaling. The Kubernetes deployment then creates a replica set to ensure that enough pods are created, with each pod running a single engine instance.

Kubernetes manages the provisioning and monitoring of the pods. The agent simply receives the status of the deployed engine instances, and communicates the status to Control Hub.

When you stop a Control Hub Kubernetes deployment, all Kubernetes resources created for that deployment are deleted.

Important: Do not directly modify the provisioned resources in your Kubernetes cluster. Doing so may cause unexpected errors.

Before you create a Control Hub Kubernetes deployment, you must complete several prerequisites.

Secrets

When you start a Control Hub Kubernetes deployment, the following information is stored as a Kubernetes secret:
  • Authentication token that the deployment uses to communicate with IBM StreamSets.
  • Proxy credentials, including the HTTP and HTTPS proxy user and password, when you configure engines to use a proxy server.

Prerequisites

Before you create a Control Hub Kubernetes deployment, complete the following prerequisites:
Create a Kubernetes environment
Create and activate a Control Hub Kubernetes environment and launch an IBM StreamSets Kubernetes agent for that environment, as described in Kubernetes Environments.
Optionally, create a Kubernetes service account
By default, each Kubernetes deployment provisioned in the Kubernetes namespace uses the default service account configured for the namespace. If you require a specific service account for this deployment, ask your Kubernetes administrator to create a service account.
You can skip this prerequisite when you want to use the default service account.
Optionally, set up an external resource archive
When your pipelines require external resources and when you plan to deploy multiple engine instances, you must set up an external resource archive that all engine instances can access. When your pipelines do not require external resources or when using a single engine instance to get started with IBM StreamSets, you do not need to complete this prerequisite.
You typically configure a deployment to use an external resource archive when you are ready to move to production, after you have finished building your pipelines and have finalized the list of external resources that your pipelines require. For more information, see External Resources.

Autoscaling

You can configure a Kubernetes deployment to automatically scale the number of pods that host engine instances based on the current processing demand.

During times of high demand, Kubernetes automatically increases the number of engine instances. For each instance, Kubernetes creates a replicated pod, and then deploys and launches a single engine instance to the pod. During times of low demand, Kubernetes automatically decreases the number of engine instances. For each instance, Kubernetes stops the engine and removes the pod.

When you enable autoscaling, you define the minimum and maximum number of pods to create. You also define the thresholds that determine when to increase or decrease the number of pods.

You can configure Full Autoscaling (Recommended) or Legacy Autoscaling. Use the recommended full autoscaling for a more robust solution during times of high demand.

The following table highlights the differences between the autoscaling types:
Functionality Full Autoscaling Legacy Autoscaling
Prerequisites Requires that the parent Kubernetes environment use Kubernetes agent version 1.3.0 or later. Requires a horizontal pod autoscaler and a metrics server set up by your Kubernetes administrator.
Metrics Scales pods based on CPU usage, memory usage, and running pipeline count. Scales pods based on CPU usage only.
Thresholds Considers a range of values to scale out and in.

For example, add more pods when CPU usage exceeds 70% and remove pods when CPU usage falls below 30%.

 Considers a single value to scale out and in.

For example, add more pods when CPU usage exceeds 50% and remove pods when CPU usage falls below 50%.
Minutes a threshold must be met before scaling pods Configurable number of minutes. Not configurable.
Determining which pod to remove when scaling in Attempts to remove pods with the fewest number of running pipelines. Arbitrarily removes pods.

Full Autoscaling (Recommended)

With full autoscaling, the Kubernetes agent automatically scales the number of pods that host engine instances based on the following metrics:
  • CPU usage - Percentage of CPU used by each pod that hosts an engine instance.
  • Memory usage - Percentage of memory used by each engine instance.
  • Running pipeline count - Number of running pipelines on each engine instance.
Important: The parent Kubernetes environment must use the Kubernetes agent version 1.3.0 or later before you can define full autoscaling for a Kubernetes deployment.

You can configure a deployment to monitor all or some of the metrics. The Kubernetes agent monitors the average of the selected metrics across all engines belonging to the deployment.

You define the thresholds that the Kubernetes agent uses to increase or decrease the number of pods that host engine instances. You can define the following types of thresholds:
Scaling out thresholds
Scaling out thresholds determine when to increase the number of pods. When the thresholds for any of the metrics are met, the Kubernetes agent increases the number of pods.
For example, you configure a deployment to monitor the CPU and running pipeline count metrics. You define the following scaling out thresholds:
  • Average CPU % higher than 70
  • Average running pipeline count higher than 3
When only one threshold is met, for example the average CPU reaches 80% but the average running pipeline count is 1, the agent increases the number of pods.
Scaling in thresholds
Scaling in thresholds determine when to decrease the number of pods. When the thresholds for all of the metrics are met, the Kubernetes agent decreases the number of pods.
For example, you configure a deployment to monitor the CPU and running pipeline count metrics. You define the following scaling in thresholds:
  • Average CPU % lower than 30
  • Average running pipeline count lower than 2
When only one threshold is met, for example the average CPU is 20% but the average running pipeline count is 2, the agent does not decrease the number of pods. When both thresholds are met, for example the average CPU is 20% and the average running pipeline count is 1, the agent decreases the number of pods.
When scaling in, the Kubernetes agent attempts to remove pods with the fewest number of running pipelines.
Note: To maintain stability, the Kubernetes agent does not immediately increase or decrease pods when the average metrics exceed or fall below the thresholds. Instead, the agent waits for the number of minutes configured in the Monitor Metrics Time property before making changes.

Legacy Autoscaling

Legacy autoscaling scales the number of pods that host engine instances based on CPU usage only.

Important: Legacy autoscaling uses a horizontal pod autoscaler and a metrics server to scale the pods. Your Kubernetes administrator must set up a running metrics or equivalent server in the Kubernetes cluster as an environment prerequisite.

With legacy autoscaling, you define the CPU threshold percentage that determines when to increase or decrease the number of pods. You also must configure either the CPU Requested or CPU Limit property, or both.

For example, if CPU Requested is 100m, and CPU Threshold Percentage is 50%, Kubernetes creates additional pods that host engine instances when the average CPU usage for all existing pods exceeds 50%. Kubernetes removes pods when the average CPU usage falls below 50%.

When scaling in, Kubernetes arbitrarily removes pods.

Note: To maintain stability, Kubernetes does not immediately increase or decrease pods when the average CPU usage exceeds or falls below the threshold. Instead, it waits for some time before making changes.

Configuring a Kubernetes Deployment

Configure a Control Hub Kubernetes deployment to define the group of engine instances to deploy to a Kubernetes environment.

Important: Before configuring a deployment, you must complete the required prerequisites.

To create a new deployment, click Set Up > Deployments in the Navigation panel, and then click the Create Deployment icon: Create Deployment icon.

To edit an existing deployment, click Set Up > Deployments in the Navigation panel, click the deployment name, and then click Edit.

Then, complete the following steps in the deployment wizard:
  1. Define the Deployment
  2. Configure the Engine
  3. Configure the Kubernetes Deployment
  4. Share the Deployment
  5. Review and Launch the Deployment

Define the Deployment

Define the deployment essentials, including the deployment name and type, the environment that the deployment belongs to, and the engine type and version to deploy.

Once saved, you cannot change the deployment type, the engine version, or the environment.

  1. Configure the following properties:
    Define Deployment Property Description
    Deployment Name Name of the deployment.

    Use a brief name that informs your team of the deployment use case.
    Deployment Type Select Kubernetes.
    Environment Active Kubernetes environment where engine instances will be deployed.
    Engine Type Type of engine to deploy:
    • Data Collector
    • Transformer
    Engine Version Engine version to deploy.
    Deployment Tags Optional tags that identify similar deployments within Control Hub. Use deployment tags to easily search and filter deployments.

    Enter nested tags using the following format:

    <tag1>/<tag2>/<tag3>
  2. If creating the deployment, click one of the following buttons:
    • Cancel - Cancels creating the deployment and exits the wizard.
    • Save & Next - Saves the deployment and continues.
    • Save & Exit - Saves the deployment and exits the wizard, displaying the incomplete deployment in the Deployments view.

Configure the Engine

Define the configuration of the engine to deploy. You can use the defaults to get started.

  1. Configure the following properties:
    Engine Property Description
    Stage Libraries

    Stage libraries to install on the engine.

    The available stage libraries depend on the selected engine type and version.
    Advanced Configuration

    Access to advanced configuration properties to further customize the engine. As you get started with StreamSets, the default values should work in most cases.

    The available properties depend on the selected engine type.
    External Resource Source Source of the external files and libraries, such as JDBC drivers, required by the engine:
    • None - External resources are not defined in the deployment.

      Select when using a single engine instance to get started with IBM StreamSets, or when your pipelines do not require external resources.

    • Archive File - External resources are included in an archive file defined in the deployment.

      Select when the deployment launches multiple engine instances and when your pipelines require external resources.
    External Resource Location

    Location of the archive file that contains the external resources used by the engine. The archive file must be in TGZ or ZIP format.

    Enter the location using one of the following formats:
    • File path. For example: /mnt/shared/externalResources.tgz
      Important: To use a file path, you must use advanced mode to edit the deployment YAML file to mount the file to the engine container.
    • URL. For example: https://<hostname>:<port>/shared/externalResources.tgz
    Tip: Click the download icon to download a sample externalResources.tgz file to view the required directory structure.

    Available when using an archive file as the source for external resources.
    Engine Labels Labels to assign to all engine instances launched for this deployment. Labels determine the group of engine instances that run a job.

    Default is the name of the deployment.
    Max CPU Load (%)

    Maximum percentage of CPU on the host machine that an engine instance can use. When an engine equals or exceeds this threshold, Control Hub does not start new pipeline instances on the engine.

    All engine instances belonging to the deployment inherit these resource threshold values.

    Default is 80.
    Max Memory (%)

    Maximum percentage of the configured Java heap size that an engine instance can use. When an engine equals or exceeds this threshold, Control Hub does not start new pipeline instances on the engine.

    Default is 100.
    Max Running Pipeline Count

    Maximum number of pipelines that can be running on each engine instance. When an engine equals this threshold, Control Hub does not start new pipeline instances on the engine.

    Default is 1,000,000.
  2. If creating the deployment, click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the deployment and continues.
    • Save & Exit - Saves the deployment and exits the wizard, displaying the incomplete deployment in the Deployments view.

Configure the Kubernetes Deployment

Configure details about the pods provisioned in the Kubernetes cluster.

  1. If you are an advanced Kubernetes user and want to directly edit the deployment YAML, select Advanced Mode.
    Note: As you configure the deployment, Control Hub generates a valid YAML file used to provision the Kubernetes resources. The automatically-generated YAML file is sufficient for most use cases.

    For details about using advanced mode, see Advanced Mode.

  2. Configure the following properties:
    Kubernetes Deployment Property Description
    Kubernetes Labels Kubernetes labels to apply to all Kubernetes resources provisioned for this deployment.
    Enter the labels as key-value pairs. For label naming requirements, see the Kubernetes documentation.
    Important: IBM StreamSets reserves app as a label key for its own use. As a result, you cannot define app as a label key.

    You can define the labels using simple or bulk edit mode. In simple edit mode, click Add Another to define additional labels. In bulk edit mode, configure labels in JSON format.

    Note: These labels are applied to Kubernetes resources, not to Control Hub deployments.
    Enable Autoscaling Automatically scale the number of pods that host engine instances based on the current processing demand.
    Autoscaling Type Type of autoscaling to use:
    • Full (Recommended) - The Kubernetes agent automatically scales the number of pods based on selected metrics which can include CPU usage, memory usage, and a running pipeline count.
    • Legacy - Kubernetes uses a horizontal pod autoscaler and a metrics server to scale the number of pods based on CPU only. Requires that your Kubernetes administrator set up a metrics server as an environment prerequisite.

      Also requires that you configure either the CPU Requested or CPU Limit property, or both.

      Note: If you edit an active Kubernetes deployment to enable legacy autoscaling, Kubernetes initially decreases the number of pods to one until it can create a horizontal pod autoscaler, which can take several minutes.
    Metrics Engine metrics used to perform full autoscaling. Select one or more of the following metrics:
    • Use CPU % - Monitors the percentage of CPU used by each pod that hosts an engine instance.
    • Use Memory % - Monitors the percentage of memory used by each engine instance.
    • Use Running Pipeline Count - Monitors the number of running pipelines on each engine instance.

    Available when full autoscaling is enabled.
    Scaling Out Thresholds that determine when to increase the number of pods that host engine instances based on the selected metrics.

    Enter a threshold value for each metric. When the thresholds for any of the metrics are met, the agent increases the number of pods.

    Available when full autoscaling is enabled.
    Scaling In Thresholds that determine when to decrease the number of pods that host engine instances based on the selected metrics.

    Enter a threshold value for each metric. When the thresholds for all of the metrics are met, the agent decreases the number of pods.

    Available when full autoscaling is enabled.
    Monitor Metrics Time (minutes) Number of minutes a threshold must be met before the Kubernetes agent scales pods that host engine instances. The agent monitors the metrics every minute.

    Available when full autoscaling is enabled.
    Cool Down Period (minutes) Number of minutes after a scaling event that the Kubernetes agent does not perform additional scaling, even when a threshold is met.

    Available when full autoscaling is enabled.
    Minimum Instances Minimum number of engine instances to deploy.

    Minimum value is 1.

    Available when either autoscaling type is enabled.
    Maximum Instances Maximum number of engine instances to deploy.

    Available when either autoscaling type is enabled.
    CPU Threshold Percentage Target average CPU utilization, represented as a percentage of the requested CPU, over all pods that host an engine instance.

    Kubernetes creates additional pods that host engine instances when the average CPU usage for all existing pods exceeds this percentage. Kubernetes removes pods when the average CPU usage falls below this percentage.

    Available when legacy autoscaling is enabled.
    Desired Instances Number of engine instances to deploy.
    For each instance, Kubernetes creates a replicated pod, and then deploys and launches a single engine instance to the pod.
    Important: If your pipelines require external resources, you must set up an external resource archive that all engine instances can access before increasing the number of instances.

    Default is 1. Set to the minimum value of 0 to temporarily prevent engine instances from running, as an alternative to stopping the deployment but that still incurs minimal costs from the cloud service provider.

    Available when autoscaling is disabled.
    CPU Requested Requested amount of CPU for each pod hosting an engine instance.

    A pod is guaranteed to have as much CPU as it requests.
    CPU Limit Maximum amount of CPU that each pod hosting an engine instance can use.
    Memory Requested Requested amount of memory for each pod hosting an engine instance.

    Include the units when you enter a value. For example, enter 1024Mi to specify 1024 mebibytes. For more information about Kubernetes memory resource units, see the Kubernetes documentation.

    A pod is guaranteed to have as much memory as it requests.
    Memory Limit Maximum amount of memory in megabytes that each pod hosting an engine instance can use.
    Service Account Name Name of the Kubernetes service account to associate with the Kubernetes deployment provisioned in the namespace. Specify when your Kubernetes administrator has created a service account as a prerequisite.

    When not specified, the default service account configured for the namespace is used.
  3. If creating the deployment, click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the deployment and continues.
    • Save & Exit - Saves the deployment and exits the wizard, displaying the incomplete deployment in the Deployments view.

Share the Deployment

By default, the deployment can only be seen by you. Share the deployment with other users and groups to grant them access to it.

  1. In the Select Users and Groups field, type a user email address or a group name.
  2. Select users or groups from the list, and then click Add.

    The added users and groups display in the User / Group table.

  3. Modify permissions as needed. By default, each added user or group is granted the following permissions:
    • Read - View the details of the deployment and of all engines managed by the deployment. Restart or shut down individual engines managed by the deployment in the Engines view.
    • Write - Edit, start, stop, and delete the deployment. Delete engines managed by the deployment. Also requires read access on the parent environment.
    • Execute - Start jobs on engines managed by the deployment. Starting jobs also requires execute access on the job and read access on the pipeline.

    For more information, see Deployment Permissions.

  4. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the deployment and continues.
    • Save & Exit - Saves the deployment and exits the wizard, displaying the incomplete deployment in the Deployments view.

Review and Launch the Deployment

You've successfully finished creating the deployment.

  1. Click one of the following buttons:
    • Exit - Saves the deployment and exits the wizard, displaying the Deactivated deployment in the Deployments view. You can start the deployment at a later time.
    • Launch Deployment - Starts the deployment, as long as the IBM StreamSets Kubernetes agent that corresponds to the parent environment is online. The agent communicates with Control Hub to provision the Kubernetes resources needed to run engines and to deploy engine instances to those resources.
      Note: In some cases, Kubernetes can take several minutes to create and run all resources. A Control Hub Kubernetes deployment transitions to an Active state only when all associated Kubernetes resources are running.
  2. If the deployment launches a Transformer engine that works with a Spark cluster, you must grant the Spark cluster access to Transformer.

    For instructions, see Granting the Spark Cluster Access to Transformer in the Transformer engine documentation.

Advanced Mode

As you configure a Kubernetes deployment, Control Hub generates a valid YAML file used to provision the Kubernetes resources. The automatically generated YAML file is sufficient for most use cases. However, if you are an advanced Kubernetes user, you can use advanced mode to directly edit the deployment YAML file. For example, you might want to edit the YAML to attach extra volumes or to use a custom image.

To access advanced mode, in the Configure Kubernetes Deployment step in the deployment wizard, select Advanced Mode.

The wizard displays the generated YAML. You can directly edit the YAML in the wizard. Or, click the Download file icon (Download file icon) to edit the YAML in a text editor and then upload the edited file.

Click the Reset icon (Reset icon) to reset to the previously saved YAML.

Note: When you edit a cloned Kubernetes deployment, you can select Show Diff to display the YAML differences between the original and cloned deployment.

Use caution when editing the YAML. Control Hub validates that the YAML uses the correct syntax, but cannot validate that you have specified an existing volume or image. Control Hub does place some restrictions on the edits you can make to the file.

The maximum YAML size is 16 KB.

Important: If you add custom objects in the advanced YAML, ensure that the Kubernetes agent has sufficient permissions to apply the custom YAML.

Editing a Kubernetes Deployment

You can edit a Control Hub Kubernetes deployment while it is deactivated or active.

When you stop a Control Hub Kubernetes deployment, all Kubernetes resources created for that deployment are deleted. After you edit properties and then restart the deployment, the IBM StreamSets Kubernetes agent communicates with Control Hub to provision the Kubernetes resources needed to run engines and to deploy engine instances to those resources.

When you edit a deployment while it is active, existing Kubernetes resources might be deleted, depending on the following types of edited properties:

General deployment or engine properties
When you edit general deployment or engine properties while the deployment is active, the Kubernetes agent continues running the existing pods. Changes are replicated to all engine instances on the next restart of the engines.
For example, let's say you edit the deployment to install additional stage libraries on the engine instances, and then you instruct Control Hub to restart all engine instances. The Kubernetes agent restarts the engine instances on the existing pods, which triggers the installation of the additional stage libraries and the engine property changes.
Kubernetes properties
When you edit Kubernetes properties while the deployment is active, Kubernetes might replace all of the existing Kubernetes pods, depending on the change. If a replacement is needed, Kubernetes deletes the pods one by one to prevent engine downtime.
For example, if you edit the deployment to increase the number of engine instances from 2 to 3, the Kubernetes agent applies the changes and Kubernetes provisions a new pod. If you edit a deployment to enable legacy autoscaling, the Kubernetes agent creates a horizontal pod autoscaler, which might delete the existing Kubernetes pods or provision new ones.

To edit a deployment, locate the deployment in the Deployments view. In the Actions column, click the More icon (More icon) and then click Edit.