Kubernetes Environments

1. In the Navigation panel, click Set Up > Environments, click the Kubernetes environment name, and then click Edit.
2. For the Feature Version property, select a later feature version and then click Save.

   Control Hub restarts all deployments belonging to the environment.

3. Optionally, an organization administrator can update the default feature version to use for all new Kubernetes environments.

1. Configure the following properties:

   Define Environment Property	Description
   Environment Name	Name of the environment. Use a brief name that informs your team of the environment use case.
   Environment Type	Select Kubernetes. Once saved, you cannot change the environment type.
   Environment Tags	Optional tags that identify similar environments within Control Hub. Use environment tags to easily search and filter environments. Enter nested tags using the following format: <tag1>/<tag2>/<tag3>
   Feature Version	Feature version to use for the environment and all deployments created for the environment. Each feature version typically requires different permissions or a restart of engines belonging to the deployments. When creating a new environment, use the latest feature version. When a new feature version is available, change your existing environments to use the new feature version as soon as possible.

2. Optionally, click Show Advanced Options and configure the following advanced property:

   Define Environment Advanced Property

   Description

   Allow Nightly Builds

   Allows deployments for this environment to use nightly engine builds in addition to released engine versions. Also allows Kubernetes environments to use nightly IBM StreamSets Kubernetes agent builds. Nightly builds are for testing features under development and should not be used in production systems. The version number of a nightly build includes a -SNAPSHOT suffix and the build number. For example, 5.2.0-SNAPSHOT (Build 1013).

3. If creating the environment, click one of the following buttons:
   • Cancel - Cancels creating the environment and exits the wizard.
   • Save & Next - Saves the environment and continues.
   • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

1. Configure the following properties:

   Configure Environment Property	Description
   Agent Version	Version of the Kubernetes agent to deploy. As a best practice, use the latest released version.
   Kubernetes Namespace	Name of the Kubernetes namespace where you plan to deploy IBM StreamSets engines. Enter the name of the namespace created as a prerequisite by your Kubernetes administrator. You cannot change the namespace name while the environment is active.
   Agent Java Options	Custom Java options used by the agent. Enter any options supported by the JVM.
   Kubernetes Labels	Kubernetes labels to apply to all Kubernetes resources provisioned for this environment. Enter the labels as key-value pairs. For label naming requirements, see the Kubernetes documentation. Important: IBM StreamSets reserves app as a label key for its own use. As a result, you cannot define app as a label key. You can define the labels using simple or bulk edit mode. In simple edit mode, click Add Another to define additional labels. In bulk edit mode, configure labels in JSON format. Note: These labels are applied to Kubernetes resources, not to Control Hub environments.

2. If creating the environment, click one of the following buttons:
   • Back - Returns to the previous step in the wizard.
   • Save & Next - Saves the environment and continues.
   • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

1. In the Select Users and Groups field, type a user email address or a group name.
2. Select users or groups from the list, and then click Add.

   The added users and groups display in the User / Group table.

3. Modify permissions as needed. By default, each added user or group is granted the following permissions:
   • Read - View the details of the environment. Create and edit a deployment for the environment.
   • Write - Edit, activate, deactivate, and delete the environment.

   For more information, see Environment Permissions.

4. Click one of the following buttons:
   • Back - Returns to the previous step in the wizard.
   • Save & Next - Saves the environment and continues.
   • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

1. Click Activate & Generate Install Script to activate the environment and generate an installation script and YAML file for the IBM StreamSets Kubernetes agent.
   Note: If you click Exit, Control Hub saves the environment and exits the wizard, displaying the deactivated environment in the Environment view. You can activate the environment and retrieve the installation script or YAML file at a later time.
2. To launch the agent from the installation script, click the Copy to Clipboard icon () to copy the generated installation script.
3. To launch the agent from the YAML file, complete the following steps:
   1. Click View Kubernetes YAML.
   2. Click the Copy to Clipboard icon () to copy the YAML content, and then click Close.
   3. Save the YAML to your local file system.
4. Optionally, click Check Agent Status after Install to display the Agent Status window, where you can view the agent status after you launch the agent.
5. Use the copied installation script or YAML file to launch the Kubernetes agent.

You can launch the IBM StreamSets Kubernetes agent using the installation script that Control Hub generates. The installation script automatically retrieves the agent YAML file from Control Hub, and then uses the Kubernetes command-line tool, kubectl, to create a deployment for the agent based on the YAML file.

1. Verify that you have completed the kubectl prerequisites.
2. Open a command prompt.
3. To verify that kubectl can access your Kubernetes cluster, run the following command:

   kubectl cluster-info

4. Paste and then run the installation script command that you copied from the Control Hub Kubernetes environment.
   Note: If needed, you can retrieve the generated installation script.
5. If you chose to check the agent status, view the status in the Control Hub UI.

You can launch the IBM StreamSets Kubernetes agent using the YAML file that Control Hub generates. Use the Kubernetes command-line tool, kubectl, to create a deployment for the agent based on the YAML file, as you do with other Kubernetes applications.

1. Verify that you have completed the kubectl prerequisites.
2. Open a command prompt.
3. To verify that kubectl can access your Kubernetes cluster, run the following command:

   kubectl cluster-info

4. Run the following command to create a deployment for the YAML file that you copied from the Control Hub Kubernetes environment:

   kubectl apply -f ./<file_name>.yaml

   For example, if you saved the YAML to a file named streamsets-kubernetes-agent.yaml, run the following command:

   kubectl apply -f ./streamsets-kubernetes-agent.yaml

   Note: If needed, you can retrieve the generated YAML file.
5. If you chose to check the agent status, view the status in the Control Hub UI.

You can retrieve the installation script generated for a Kubernetes environment.

1. In the Control Hub Navigation panel, click Set Up > Environments.
2. Locate the Kubernetes environment that you want to launch an agent for.
3. In the Actions column, click the More icon () and then click Get Install Script.
4. Click the Copy to Clipboard icon (), and then click Close.

You can retrieve the YAML file generated for a Kubernetes environment.

1. In the Control Hub Navigation panel, click Set Up > Environments.
2. Locate the Kubernetes environment that you want to launch an agent for.
3. In the Actions column, click the More icon () and then click View Generated YAML.
4. Click the Copy to Clipboard icon (), and then click Close.

1. In the Control Hub Navigation panel, click Set Up > Environments.
2. In the Actions column of the Kubernetes environment, click the More icon () and then click View Generated YAML.
3. Click the Copy to Clipboard icon (), and then click Close.
4. Save the YAML to your local file system.
5. Edit the YAML to add the proxy properties as Java options to the STREAMSETS_KUBERNETES_AGENT_JAVA_OPTS environment variable:

   - name: STREAMSETS_KUBERNETES_AGENT_JAVA_OPTS
     value: -Dhttps.proxyHost=<proxy host> -Dhttps.proxyPort=<port> -Dhttp.proxyHost=<proxy host> -Dhttp.proxyPort=<port> -Dhttp.nonProxyHosts=<pipe-separated no proxy hosts> -Dhttp.proxyUser=<proxy user> -Dhttp.proxyPassword=<password> -Dhttps.proxyUser=<proxy user> -Dhttps.proxyPassword=<password> -Djdk.http.auth.tunneling.disabledSchemes=

   For the -Dhttp.nonProxyHosts option, include the ClusterIP of the Kubernetes API server and any other hosts that the agent can connect to without going through the proxy server. Separate each entry with the pipe character (|).

   To retrieve the ClusterIP of the Kubernetes API server, run the following command:

   kubectl get svc/kubernetes --namespace=default

   Note: Define the user and password properties only when the proxy server requires authentication.

   The properties are the same proxy properties that you configure for IBM StreamSets engines. For a description of each property, see the Data Collector documentation or the Transformer documentation.

6. Run the following command to apply the changes to the running Kubernetes agent:

   kubectl apply -f ./<file_name>.yaml

1. In the Control Hub Navigation panel, click Set Up > Deployments.
2. Select the active deployment that belongs to the Kubernetes environment, and then click the Stop icon: .
3. Click OK to confirm, and then click Close.
4. In the Actions column of the deactivated deployment, click the More icon () and then click Edit.
5. In the Edit Deployment dialog box, expand the Configure Deployment section.
6. Click Click here to configure next to the Advanced Configuration property.
7. In the Proxy tab, define the engine proxy properties as needed.

   For a description of each proxy property, see the Data Collector documentation or the Transformer documentation.

8. Click Save.
9. In the Edit Deployment dialog box, click Save.
10. Start the deployment to launch new engine instances that use the updated proxy properties.

    For detailed instructions, see Starting Deployments.

If the agent log file does not provide enough troubleshooting information, you can modify the log level to display messages at another severity level.

1. In the Control Hub Navigation panel, click Set Up > Environments.
2. In the Actions column of the Kubernetes environment, click the More icon () and then click View Generated YAML.
3. Click the Copy to Clipboard icon (), and then click Close.
4. Save the YAML to your local file system.
5. Add a ConfigMap section to the end of the YAML file, setting the log level that you want to use.

   For example, to set all messages to the DEBUG log level, add the following ConfigMap:

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: custom-agent-log4j
   data:
     custom_log4j_file_name: "custom-agent-log4j.properties"
     custom-agent-log4j.properties: |
       name = streamsets-dpm-kubernetes-agent
       appender.stdout.type = Console
       appender.stdout.name = stdout
       appender.stdout.layout.type = PatternLayout
       appender.stdout.layout.pattern = %d{ISO8601} [thread:%t] %-5p %c{1} - %m%n
   
       appender.streamsets.type = RollingFile
       appender.streamsets.name = streamsets
       appender.streamsets.fileName = ${sys:streamsets.kubernetes.agent.log.dir}/agent.log
       appender.streamsets.filePattern = ${sys:streamsets.kubernetes.agent.log.dir}/agent.%i.log
       appender.streamsets.layout.type = PatternLayout
       appender.streamsets.layout.pattern = %d{ISO8601} [thread:%t] %-5p %c{1} - %m%n
       appender.streamsets.append = true
       appender.streamsets.policies.type = Policies
       appender.streamsets.policies.size.type = SizeBasedTriggeringPolicy
       appender.streamsets.policies.size.size = 256MB
       appender.streamsets.strategy.type = DefaultRolloverStrategy
       appender.streamsets.strategy.max = 10
   
       rootLogger.level = DEBUG
       rootLogger.appenderRef.stdout.ref = stdout
   
       logger.l1.name = com.streamsets
       logger.l1.level = DEBUG
       logger.l1.appenderRef.streamsets.ref = streamsets

6. To define a volume for the ConfigMap and mount it from the agent container, add the following lines in bold to the Deployment section of the YAML file:

   apiVersion: apps/v1
   kind: Deployment
   ...
       spec:
         containers:
           - name: streamsets-agent
             image: icr.io/streamsets/kubernetes-agent:1.4.0
             ...
             resources:
               requests:
                 memory: 1Gi
                 cpu: 0.5
             volumeMounts:
             - name: config
               mountPath: "/opt/streamsets-kubernetes-agent/etc/"
               readOnly: true
         dnsPolicy: Default
         serviceAccountName: streamsets-agent-sa-f00bb553-59d8-48c5-b681-5bec6eb4f430
   volumes:
      - name: config
           configMap:
             name: custom-agent-log4j
             items:
             - key: "custom-agent-log4j.properties"
               path: "custom-agent-log4j.properties"

7. Add the path to the log configuration file as a Java option to the STREAMSETS_KUBERNETES_AGENT_JAVA_OPTS environment variable:

   - name: STREAMSETS_KUBERNETES_AGENT_JAVA_OPTS
     value: -Dlog4j2.configurationFile=/opt/streamsets-kubernetes-agent/etc/custom-agent-log4j.properties

8. Run the following command to apply the changes to the running Kubernetes agent:

   kubectl apply -f ./<file_name>.yaml

When you deactivate a Kubernetes environment, the IBM StreamSets Kubernetes agent is automatically deleted from the namespace for you. However, in rare situations, the agent can become unresponsive and you must retrieve and run an uninstallation script to shut down and delete the agent from the Kubernetes namespace.

1. In the Control Hub Navigation panel, click Set Up > Environments.
2. Locate the Kubernetes environment with the unresponsive agent.

   Each environment generates a unique uninstallation script for the agent as defined for that specific environment. Be sure that you retrieve the script for the correct environment.

3. In the Actions column, click the More icon () and then click Get Uninstall Script.
4. Click the Copy to Clipboard icon (), and then click Close.
5. Open a command prompt.
6. To verify that kubectl can access your Kubernetes cluster, run the following command:

   kubectl cluster-info

7. Paste and then run the uninstallation script command that you copied from the Control Hub Kubernetes environment.