Installing IBM StreamSets

An instance administrator can install IBM StreamSets on IBM Software Hub Version 5.1.

5.1.0 Requires a separate installation process. For more information, see Installing IBM StreamSets as client-managed software in the IBM StreamSets documentation.

5.1.1 and later Install IBM StreamSets in the same way that you install any other service.

Who needs to complete this task?

Instance administrator To install IBM StreamSets, you must be an instance administrator. An instance administrator has permission to install software in the following projects:

The operators project for the instance

The operators for this instance of IBM StreamSets are installed in the operators project.

In the installation commands, the ${PROJECT_CPD_INST_OPERATORS} environment variable refers to the operators project.

The operands project for the instance

The custom resources for the control plane and IBM StreamSets are installed in the operands project.

In the installation commands, the ${PROJECT_CPD_INST_OPERANDS} environment variable refers to the operands project.

When do you need to complete this task?

Review the following options to determine whether you need to complete this task:

  • If you want to install multiple services at the same time, follow the process in Running a batch installation of solutions and services instead.
  • If you didn't install IBM StreamSets as part of a batch installation, complete this task to add IBM StreamSets to your environment.

    Repeat as needed If you are responsible for multiple instances of IBM Software Hub, you can repeat this task to install more instances of IBM StreamSets on the cluster.

Information you need to complete this task

Review the following information before you install IBM StreamSets:

Version requirements

All of the components that are associated with an instance of IBM Software Hub must be installed at the same release. For example, if the IBM Software Hub control plane is installed at Version 5.1.3, you must install IBM StreamSets at Version 5.1.3.

Environment variables

The commands in this task use environment variables so that you can run the commands exactly as written.

  • If you don't have the script that defines the environment variables, see Setting up installation environment variables.
  • To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:
    source ./cpd_vars.sh
Security context constraint

IBM StreamSets works with the default Red Hat® OpenShift® Container Platform security context constraint, restricted-v2.

Storage requirements
You must specify storage classes when you install IBM StreamSets. The following storage classes are recommended. However, if you don't use these storage classes on your cluster, ensure that you specify a storage class with an equivalent definition.
Storage Notes Storage classes
OpenShift Data Foundation 5.1.3 and later IBM StreamSets supports OpenShift Data Foundation in IBM Software Hub Version 5.1.3 and later only.

When you install the service, specify file storage.
  • File storage: ocs-storagecluster-cephfs
IBM Fusion Data Foundation Not supported. Not applicable.
IBM Fusion Global Data Platform Not supported. Not applicable.
IBM Storage Scale Container Native Not supported. Not applicable.
Portworx
  • 5.1.0 On IBM Software Hub Version 5.1.0, ensure that a default storage class is configured on the cluster before you install IBM StreamSets. When you install IBM StreamSets, it uses the default storage class on the cluster.
  • 5.1.1 and later On IBM Software Hub Version 5.1.1, Version 5.1.2, or Version 5.1.3, specify file storage when you install the service.
  • File storage: portworx-shared-gp3
NFS Not supported. Not applicable.
Amazon Elastic storage
  • 5.1.0 On IBM Software Hub Version 5.1.0, ensure that a default storage class is configured on the cluster before you install IBM StreamSets. When you install IBM StreamSets, it uses the default storage class on the cluster.
  • 5.1.1 and later On IBM Software Hub Version 5.1.1, Version 5.1.2, or Version 5.1.3, specify file storage when you install the service.

File storage is provided by Amazon Elastic File System.
  • File storage: efs-nfs-client
NetApp Trident Not supported. Not applicable.
Nutanix Not supported. Not applicable.

Before you begin

This task assumes that the following prerequisites are met:

Prerequisite Where to find more information
The cluster meets the minimum requirements for installing IBM StreamSets. If this task is not complete, see System requirements.
The workstation from which you will run the installation is set up as a client workstation and includes the following command-line interfaces:
  • IBM Software Hub CLI: cpd-cli
  • OpenShift CLI: oc
 If this task is not complete, see Setting up a client workstation.
The IBM Software Hub control plane is installed. If this task is not complete, see Installing an instance of IBM Software Hub.
For environments that use a private container registry, such as air-gapped environments, the IBM StreamSets software images are mirrored to the private container registry. If this task is not complete, see Mirroring images to a private container registry.
For environments that use a private container registry, such as air-gapped environments, the cpd-cli is configured to pull the olm-utils-v3 image from the private container registry. If this task is not complete, see Pulling the olm-utils-v3 image from the private container registry.

Procedure

Complete the following tasks to install IBM StreamSets:

  1. Installing the service
  2. Validating the installation
  3. What to do next

Installing the service

To install IBM StreamSets:

  1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:
    ${CPDM_OC_LOGIN}
    Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
  2. Run the following command to create the required OLM objects for IBM StreamSets in the operators project for the instance:
    cpd-cli manage apply-olm \
--release=${VERSION} \
--cpd_operator_ns=${PROJECT_CPD_INST_OPERATORS} \
--components=streamsets
    Wait for the cpd-cli to return the following message before you proceed to the next step:
    [SUCCESS]... The apply-olm command ran successfully

    If the apply-olm fails, see Troubleshooting the apply-olm command during installation or upgrade.

  3. Create the custom resource for IBM StreamSets.

    The command that you run depends on the storage on your cluster.

    Red Hat OpenShift Data Foundation storage

    Run the following command to create the custom resource.

    cpd-cli manage apply-cr \
--components=streamsets \
--release=${VERSION} \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--block_storage_class=${STG_CLASS_BLOCK} \
--file_storage_class=${STG_CLASS_FILE} \
--license_acceptance=true
    IBM Fusion Data Foundation storage

    Run the following command to create the custom resource.

    cpd-cli manage apply-cr \
--components=streamsets \
--release=${VERSION} \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--block_storage_class=${STG_CLASS_BLOCK} \
--file_storage_class=${STG_CLASS_FILE} \
--license_acceptance=true
    Portworx storage

    Run the following command to create the custom resource.

    cpd-cli manage apply-cr \
--components=streamsets \
--release=${VERSION} \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--storage_vendor=portworx \
--license_acceptance=true
    AWS with EFS and EBS storage

    Run the following command to create the custom resource.

    cpd-cli manage apply-cr \
--components=streamsets \
--release=${VERSION} \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--block_storage_class=${STG_CLASS_BLOCK} \
--file_storage_class=${STG_CLASS_FILE} \
--license_acceptance=true

Validating the installation

IBM StreamSets is installed when the apply-cr command returns:
[SUCCESS]... The apply-cr command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=streamsets

What to do next

You must complete Post-installation setup for IBM StreamSets before users can access IBM StreamSets.