Upgrade a Data Collector engine

About this task

To upgrade to a later Data Collector engine version, create a new StreamSets environment that uses the later engine version and then create new flows or upgrade existing flows to use the new engine version.

Procedure

  1. Create a new StreamSets environment that uses the later engine version.
    1. On the Manage tab of your project, click the StreamSets tool, and then click New environment.
    2. Select the later Data Collector engine version.
    3. In the Configure details section, use the same configurations as the existing environment.

      For example, select the same stage libraries and the same external resource archive file.

    4. If you plan to run the new and existing engines on the same workstation at the same time, expand the Advanced configuration section and define a unique value for the https.port Data Collector engine property.

      For more information about defining the https.port property, see Configure Data Collector engine properties.

    5. If an administrator for your account has configured StreamSets engines to use the direct communication method, and if you plan to run the new and existing engines on the same workstation at the same time, expand the Advanced configuration section and define a unique value for the https.port Data Collector engine property.

      For more information about the communication methods, see Engine communication.

      For more information about defining the https.port property, see Configure Data Collector engine properties.

    6. Click Save.
  2. Run the engine for the new environment.

    For more information, see Running the engine command.

  3. Create new flows that use the later engine version.

    When you create a new flow, select the environment for the later engine version. You can use the new features available in the later engine version as you build the flow.

  4. Upgrade existing flows to use the later engine version.
    1. Open a flow in the canvas and click the Settings icon.
    2. In the Flow settings dialog box, on the Run tab, select the environment for the later engine version.
    3. Click Save.

      IBM watsonx.data integration upgrades the flow so that the flow can use the new features available in the later engine version.

    4. Review the engine post upgrade tasks to complete any required tasks after the upgrade.
    5. Repeat these steps for each existing flow that you want to upgrade.
  5. Upgrade existing jobs to use the later engine version.
    1. Access the job details by clicking the job name on the Assets or Jobs tab.
    2. If the job is running, select the Options menu for the job run and click Cancel. Then, click Cancel run to confirm your request.
    3. Click the Edit Configuration icon, and then click Next on the Define details tab.
    4. On the Settings tab, select the environment for the later engine version.
    5. Click Save.

      Watsonx.data integration upgrades the job so that the job can use the new features available in the later engine version.

    6. Repeat these steps for each existing job that you want to upgrade.
  6. Delete the environment created for the previous engine version.

    For more information, see Deleting an environment.