IBM StreamSets as Client-Managed Software

A Red Hat OpenShift Container Platform cluster administrator uses the command line to install IBM StreamSets on IBM Software Hub version 5.1.0.

1. Download and install the ibm-pak plug-in.

   Download the plug-in from the IBM Catalog Management Plug-in for IBM Cloud Paks repository on GitHub. For installation instructions, see the Readme file.

2. Download the IBM StreamSets package:

   oc ibm-pak get ibm-streamsets --version 1.0.0+20250114.114609.21

3. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

4. Accept the IBM StreamSets license:

   export LICENSE_ACCEPTANCE=true

5. Install the IBM StreamSets operator. If you purchased the production license, set the production argument to true. If you purchased the non-production license, set the argument to false:

   oc ibm-pak launch ibm-streamsets --version 1.0.0+20250114.114609.21 --inventory streamsetsOperator --namespace ${PROJECT_CPD_INST_OPERATORS} --action install --args "--production <false|true>"

6. Verify that the IBM StreamSets operator is running:

   kubectl get pods -n ${PROJECT_CPD_INST_OPERATORS} | grep streamsets

7. Set the namespace where you want to install the IBM StreamSets operands:

   kubectl config set-context --current --namespace=${PROJECT_CPD_INST_OPERANDS}

8. Install the IBM StreamSets operands, setting the production argument based on your license:

   oc ibm-pak launch ibm-streamsets --version 1.0.0+20250114.114609.21 --inventory streamsets --namespace ${PROJECT_CPD_INST_OPERANDS} --action applyCustomResources --args "--production <false|true>"

9. Verify that the IBM StreamSets components are running:

   kubectl get pods -l icpdsupport/addOnId=streamsets

IBM StreamSets requires an email address for each user account.

1. Log in to IBM Software Hub.
2. From the navigation menu, select Administration > Access control.
3. For each user account that requires access to IBM StreamSets:
   1. Click the username.
   2. Click Edit.
   3. Verify that the user account has an associated email address. If needed, add an address.
   4. Click Save.

An IBM Software Hub instance administrator must grant users access to the IBM StreamSets service.

1. Log in to IBM Software Hub.
2. From the navigation menu, select Services > Instances.
3. Locate the streamsets instance.
4. From the action menu, select Manage access.
5. Add users and user groups.
   1. Click Add users.
   2. Select the user account that serves as the IBM StreamSets system administrator, and choose the Admin role.
   3. Select more users and user groups, and choose the User role.
   4. Click Add.
6. Inform the system administrator that they can access IBM StreamSets on the IBM Software Hub platform and complete the next post-installation task.

The IBM StreamSets system administrator must create an organization before users can log in to IBM StreamSets.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Add New Organization.
4. Enter an organization name.
5. Select main for the instance.
6. Enter the email address of the primary organization administrator.

   The primary organization administrator must be an existing IBM Software Hub user account that has this email address and that has access to the IBM StreamSets service with the User role.

7. Select Enterprise for the Account Type property.
8. Accept the defaults for the remaining properties.
9. Click Create Organization.
10. Inform the organization administrator that they can access IBM StreamSets on the IBM Software Hub platform and complete the remaining post-installation tasks.

If you have the IBM StreamSets Cartridge for Apache Spark Add On license, the IBM StreamSets system administrator must enable the Transformer engine and accept the license agreement.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Organizations.
4. Click Visit Instance in the Actions column.
5. Click Manage > Engine Versions.
6. Click the Create Engine Version icon .
7. Configure the engine version properties as follows:

   Property	Value
   Engine Type	Transformer
   Engine Version	6.1.0
   Scala Binary Version	2.12
   Build Number	Released
   Download URL	https://archives.streamsets.com/transformer/6.1.0/2.12

8. Click Save.
9. Review the license agreement, and then click Accept.

The organization administrator must invite users to the organization.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   IBM StreamSets Control Hub opens in a new browser tab.

3. In the toolbar, click the My Account icon and then click Invite Users.
4. In the Added Users window, type or paste one or more email addresses.

   The email addresses must be associated with existing IBM Software Hub user accounts that have access to the IBM StreamSets service with the User role.

5. Optionally, select one or more existing groups to add the users to.

   Control Hub provides a default all group that includes every user in the organization. To create more groups, see Creating Groups.

6. Optionally, modify the roles that are assigned to the users.

   Default role assignments for new users permit most tasks to encourage development and testing. Change the role assignments as needed to secure the integrity of your organization and data.

   When your organization uses projects, you can assign roles at the organization level or at the project level. For more information, see Projects Overview.

   For a description of each role, see Role Descriptions.

7. Click Invite.
8. Inform the users that they can access IBM StreamSets on the IBM Software Hub platform.
   Note: Users must join the organization within seven days. Otherwise, the invite expires and you must renew the invites.

A Red Hat OpenShift Container Platform cluster administrator uses the command line to upgrade IBM StreamSets from version 5.1.0 to version 5.1.1.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

2. Back up the root database credentials.
   1. Retrieve the database secret:

      kubectl get secret ibm-streamsets-db-service-account -o yaml -n ${PROJECT_CPD_INST_OPERANDS} | jq -r '.data.mysql-root-password' | pbcopy 

   2. Save the returned secret for later use.
3. Scale down the IBM StreamSets operator manager:

   kubectl patch deployment ibm-streamsets-operator-controller-manager -n ${PROJECT_CPD_INST_OPERATORS} --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value":0}]'

4. Verify that the IBM StreamSets pods are not running:

   kubectl get pods -n ${PROJECT_CPD_INST_OPERATORS} | grep streamsets | grep manager

5. Uninstall the IBM StreamSets services:

   helm uninstall aster-security-sample aster-security-ui-sample banner-sample connection-sample docs-sample jobrunner-sample messaging-sample metering-sample misc-resources-sample mysql-sample notification-sample pipelinestore-sample provisioning-sample sch-istio-sample scheduler-sample security-sample sequencing-sample sla-sample static-resources-sample timeseries-sample topology-sample tunneling-sample

6. Delete the IBM StreamSets operator components:

   kubectl delete -n ${PROJECT_CPD_INST_OPERATORS} subscription.operators.coreos.com/ibm-streamsets-operator-subscription

   kubectl delete -n ${PROJECT_CPD_INST_OPERATORS} clusterserviceversion.operators.coreos.com/ibm-streamsets-operator.v5.1.0

   kubectl delete -n ${PROJECT_CPD_INST_OPERATORS} catalogsource.operators.coreos.com/ibm-streamsets-catalog

7. Verify that the operator components are removed:

   kubectl get subscription,csv,deployment,catalogsource -n ${PROJECT_CPD_INST_OPERATORS} | grep ibm-stream

8. Uninstall the Istio command-line tool:

   istioctl uninstall -y --purge

9. Create a backup of the IBM StreamSets database persistent volume (PV), based on the storage class that you use.
10. Install IBM StreamSets on IBM Software Hub version 5.1.1.

    For more information, see Installing IBM StreamSets in the IBM Software Hub documentation.

11. Scale down the IBM StreamSets operator manager:

    kubectl patch deployment ibm-streamsets-operator-controller-manager -n ${PROJECT_CPD_INST_OPERATORS} --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value":0}]'

12. Uninstall most of the IBM StreamSets services:

    helm uninstall aster-security aster-security-ui banner connection docs jobrunner messaging metering notification pipelinestore provisioning sch-istio scheduler security sequencing sla software-misc-resources timeseries topology tunneling

13. Scale down the database StatefulSet:

    kubectl patch statefulset ibm-streamsets-mysql-deploy -n ${PROJECT_CPD_INST_OPERANDS} --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value":0}]'

14. Update the PersistentVolumeClaim (PVC) installed with version 5.1.1 to reference the previous database persistent volume (PV).
    1. Retrieve all persistent volumes in the operand namespace:

       kubectl get pv -n ${PROJECT_CPD_INST_OPERANDS}

    2. Locate the name of the database PV that has the claim ${PROJECT_CPD_INST_OPERANDS}/data-ibm-streamsets-mysql-deploy-0 and that has a Released state.

       For example, the name might be pvc-ceada801-e09a-4202-90ec-7c8941c6fcb6

    3. Retrieve a reference to the PVC created with the 5.1.1 installation:

       kubectl get pvc data-ibm-streamsets-mysql-deploy-0 -o yaml -n ${PROJECT_CPD_INST_OPERANDS} | pbcopy

    4. Edit the YAML file so that the uid and volumeName attributes are set to the UID and name of the previous database PV.
       For example:

       uid: ceada801-e09a-4202-90ec-7c8941c6fcb6
       ...
       volumeName: pvc-ceada801-e09a-4202-90ec-7c8941c6fcb6

    5. Save the edited YAML file in a local file system with the name scratch.yaml.
    6. Delete the PVC created by the 5.1.1 installation:

       kubectl delete pvc data-ibm-streamsets-mysql-deploy-0 -n ${PROJECT_CPD_INST_OPERANDS}

    7. Remove the claimRef from the previous database PV:

       kubectl patch pv <previous-database-PV> -n ${PROJECT_CPD_INST_OPERANDS} --type='json' -p='[{"op": "remove", "path": "/spec/claimRef", "value":1}]'

       For example:

       kubectl patch pv pvc-ceada801-e09a-4202-90ec-7c8941c6fcb6 -n ${PROJECT_CPD_INST_OPERANDS} --type='json' -p='[{"op": "remove", "path": "/spec/claimRef", "value":1}]'

    8. Apply the updated PVC so that it uses the previous database PV:

       kubectl apply -f "/<path-to-file>/scratch.yaml"

15. Patch the database secret retrieved in step 2:

    kubectl patch secret ibm-streamsets-db-service-account -n ${PROJECT_CPD_INST_OPERANDS} --type='json' -p='[{"op": "replace", "path": "/data/mysql-root-password", "value":"<retrieved-secret-value"}]'

16. Scale up the StatefulSet:

    kubectl patch statefulset ibm-streamsets-mysql-deploy -n ${PROJECT_CPD_INST_OPERANDS} --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value":1}]'

17. Scale up the IBM StreamSets operator:

    kubectl patch deployment ibm-streamsets-operator-controller-manager -n ${PROJECT_CPD_INST_OPERATORS} --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value":1}]'

18. Monitor the streamsetsinstaller/streamsets pod for completeness:

    watch kubectl get streamsetsinstaller/streamsets -n ${PROJECT_CPD_INST_OPERANDS} -o yaml

    When the installation is complete, the /status/streamsetsStatus field is listed as completed.

An organization can include multiple organization administrators, but only one primary organization administrator.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Organizations.
4. Click the internal ID of the organization that you want to edit.
5. In the list of users, find another user assigned the administrator role and then click Make Primary Admin.

An organization must be active so that users can log in as members of that organization.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Organizations.
4. Click the internal ID of the organization that you want to edit.
5. Click one of the following buttons:
   • Activate to activate the organization.
   • Deactivate to deactivate the organization.

As the system administrator, you can delete an organization. Deleting an organization permanently removes the organization, including all objects created for the organization.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Organizations.
4. Click the internal ID of the organization that you want to delete.
5. Click Delete.

As the system administrator, you can configure organization properties at a global level to affect all organizations or at an organization level to affect a specific organization.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Organizations.
4. In the Actions column for any listed organization, click Visit Instance.

   IBM StreamSets Control Hub opens and displays all existing organizations.

5. Configure organization properties in one of the following ways:
   • Global level - Click the More icon for the list of organizations, and then click Global Configuration.

     Global configurations are applied to all organizations, unless an organization administrator configured the property at the organization level.

     For example, an organization administrator sets the Default authoring engine timeout property to 8,000 milliseconds for the Northern Office organization. The system administrator then sets the same property at a global level to 6,000 milliseconds. All organizations use the modified value of 6,000 milliseconds, except for the Northern Office organization that retains the value of 8,000 configured at the organization level.

   • Organization level - Click the More icon for a specific organization, and then click Configuration.

     Organization-level configurations are applied to the selected organization only.

Control Hub sets default system limits on the number of objects that can exist in each organization. The limits protect the system from run-away scripts or unintended automation usage.

1. Log in to IBM Software Hub.
2. Open the IBM StreamSets service from the Services > Instances page.

   The IBM StreamSets system administrator tool opens in a new browser tab.

3. Click System Administration > Organizations.
4. In the Actions column for any listed organization, click Visit Instance.

   IBM StreamSets Control Hub opens and displays all existing organizations.

5. Access the organization configuration properties in one of the following ways:
   • Global level - Click the More icon for the list of organizations, and then click Global Configuration.
   • Organization level - Click the More icon for a specific organization, and then click Configuration.
6. Increase the system limit values.
   The organization configuration properties are grouped into several tabs. The following table lists the system limits that you can configure on each tab:

   Organization Configuration Tab	System Limits
   Jobrunner	Jobs Active jobs that are running concurrently
   Provisioning	Deployments Environments
   Security	Engines Groups
   Misc	API credentials per user Pipelines (including both draft and published pipelines) Pipeline versions or commits Scheduled tasks Subscriptions Topologies Topology versions or commits

An instance administrator can delete the service instance that is associated with IBM StreamSets from IBM Software Hub version 5.1.0 to 5.1.2.

1. Log in to IBM Software Hub.
2. From the navigation menu, select Services > Instances.
3. Locate the streamsets instance.
4. From the action menu, select Delete.

A Red Hat OpenShift Container Platform cluster administrator can uninstall the IBM StreamSets service from IBM Software Hub version 5.1.0 to 5.1.2.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

2. Delete the custom resource for IBM StreamSets.
   Set the version argument based on the IBM Software Hub version that you are uninstalling:

   IBM Software Hub Version	IBM StreamSets Case Version
   5.1.0	1.0.0+20250205.091628.22
   5.1.1	2.0.0+20250226.172611.112
   5.1.2	3.0.0+20250316.163057.119

   If you purchased the production license, set the production argument to true. If you purchased the non-production license, set the argument to false.

   oc ibm-pak launch ibm-streamsets --version <case-version> --inventory streamsets --namespace ${PROJECT_CPD_INST_OPERANDS} --action deleteCustomResources --args "--production <false|true>"

3. Delete the IBM StreamSets operator.

   Set the version argument based on your case version, and set the production argument based on your license:

   oc ibm-pak launch ibm-streamsets --version <case-version> --inventory streamsetsOperator --namespace ${PROJECT_CPD_INST_OPERATORS} --action uninstall --args "--production <false|true>"

4. Uninstall the IBM StreamSets services:

   helm uninstall aster-security aster-security-ui banner connection docs istio-base istio-cni istio-gateway istiod jobrunner messaging metering mysql notification pipelinestore provisioning sch-istio scheduler security sequencing sla software-misc-resources static-resources-600 static-resources-610 timeseries topology tunneling