Parameters

Use parameters in flows to set values for stage properties at run time. You can change parameter values for each job run without editing the flow, making your flows easier to manage and reuse.

To use parameters, create a parameter set for the project. A parameter set groups multiple parameters that you can reuse across flows in the same project. When you configure a flow, add the parameter set to the flow, and then reference the parameters in stage properties.

You can define a default value for each parameter, and then override the default value for each job or job run started from the flow. For example, you might create a directory parameter with a default value that points to a production directory. During testing, you can override the value to point to a test directory.

To use parameters, complete the following tasks:

Creating a parameter set

Create a parameter set to define a group of parameters that you can reuse across flows in the same project.

About this task

You can use the same parameter set across different flow types. However, StreamSets flows have the following limitations:
  • Support string parameters only.
  • Do not support double underscores (__) in a parameter set name.
  • Do not support environment variables.
  • Do not support value sets.

If a parameter set includes other parameter types, environment variables, or value sets, StreamSets flows disable those elements in the parameter set.

Procedure

  1. On the Assets tab of your project, click New asset.
  2. Select Define reusable sets of parameters.
  3. Enter a name and description, then click Create.
  4. Add parameters.
    1. Click Create parameter.
    2. Enter a name and select String for the parameter type.
    3. Optionally enter a default value for the parameter.
    4. Optionally enter help text for the parameter that explains how the parameter might be used or overridden.

      The help text displays in the flow settings after the parameter is used in a flow.

    5. Click Create.
  5. After you add all parameters, click Save.

Results

To view all parameter sets, on the Assets tab of your project, click Configurations.

Adding a parameter set to a flow

Add a parameter set to each flow where you want to use the parameters.

Procedure

  1. In the flow canvas, click the Add parameters icon {#}.
  2. Click Add parameter set.
  3. Select the parameter sets that you want to use, and then click Add.
  4. Click Return to canvas.

Using parameters in a flow

After you add a parameter set to a flow, you can use the parameters in stage properties.

Procedure

  1. In the flow canvas, open the stage where you want to use a parameter.
  2. Click the Parameterize icon {#} next to a property.
  3. Select a parameter to use, and then click Select.

    The parameter name displays as the property value with the following syntax:

    ${parameter_set_name>__<parameter_name>}

  4. To use a parameter to represent a part of a property value, append text to the parameter name.

    For example, you might use a RootDir parameter that is included in the WesternDataCenter parameter set and append the rest of the directory in the property as follows:

    ${WesternDataCenter__RootDir}/logfiles

Defining parameter values for a job

After you add parameters to a flow, define the parameter values for the job and job runs that start from this flow.

About this task

You define job parameter values in the flow settings. You can allow users to override these values each time they start a job run from the flow canvas.

You cannot override parameter values when you start a job run from the job details page. Subsequent runs from the job details page always use the job parameter values defined in the flow settings.

Procedure

  1. In the flow canvas, click the Settings icon.
  2. On the Flow settings page, click the Runtime parameters tab.
  3. Enter the job parameter values. Override default values as needed.
  4. To allow overriding parameter values each time you start a job run from the flow canvas, select Prompt before running a flow.
  5. Click Save.