Runtime values

Runtime values are values that you define outside of the flow and use for stage and flow properties. You can change the values for each flow run without having to edit the flow.

You can use runtime values for any flow property that allows the use of the expression language. You can, for example, use runtime values to represent batch sizes, timeouts, directories, and URI. You cannot use runtime values to represent fields.

You can use the following methods of passing runtime values to flows:

Runtime properties: Use runtime properties when you want to define values for multiple flow properties in a single file.; You define runtime properties in an external file, and then you call the properties from within a flow. At runtime, Data Collector loads the property values from the file. A runtime properties file can contain multiple properties.; Runtime properties are defined for the entire Data Collector - any flow can call them.
Runtime resources: Use runtime resources when you want to store common flow configuration properties in files with restricted permissions.; You define runtime resources in an external file, and then you call the resources from within a flow. You can restrict the permissions for the resource files, but any user who can create a flow can access the data stored in the files. Data Collector loads the property values from the files each time it processes a new batch. A runtime resource file can contain a single piece of information.; Runtime resources are defined for the entire Data Collector - any flow can call them.; Tip: Runtime resources can impact flow performance because Data Collector loads runtime resource files for each batch. Consider using runtime properties instead. Or, to more securely define sensitive values, consider using credential stores.

Runtime resources

Similar to runtime properties, runtime resources are values that you define in an external file and call from within a flow. But with runtime resources, you can restrict the permissions for the files to secure information.

Use runtime resources to store common configuration properties for multiple flows, like URLs for external systems. Note that any user who can create a flow can access the data stored in the resource files.

Tip: Runtime resources can impact flow performance because Data Collector loads most runtime resource files for each batch. Consider using runtime properties instead. Or, to more securely define sensitive values, use credential stores.

To implement runtime resources, perform the following steps:

Define each runtime resource.
Use an expression in the flow to call a runtime resource.

Step 1. Define runtime resources

Use the following steps to define runtime resources:

For each resource, create a text file.
A file must contain one piece of information to be used when the resource is called.
Optionally, restrict the permissions for the file.
Generally, anyone can read a file. To restrict permissions, configure the file so only the owner has read or write permissions for the file - in octals, that's 600 or 400. And the owner must be the system user that runs the Data Collector.

When you use the resource in the flow, you specify whether the file is restricted.
Add the file as an external resource for the engine, as described in Setting up external resources.

Step 2. Call the runtime resource

Use the runtime:loadResource or runtime:loadResourceRaw function to call a runtime resource. You can use runtime resources to represent information in any stage or flow property that allows the use of the expression language.

Note: In most cases, you'll use the runtime:loadResource function which trims any leading or trailing whitespace characters from the file. However, if needed, you can also use the runtime:loadResourceRaw function which includes any leading or trailing whitespace characters in the file.

To call a runtime resource, use the following syntax:

${runtime:loadResource(<file name>, <restricted: true | false>)}

For example, the following expression returns the contents of the JDBC.txt file, trimming any leading or trailing whitespace characters. The file contains a connection string and is restricted so only the owner can read the file:

${runtime:loadResource("JDBC.txt", true)}

Runtime properties

Runtime properties are properties that you define in an external location and call from within a flow.

With runtime properties, you can define different sets of values for different StreamSets environments. Runtime properties allow you to easily update the property definitions without editing the flow.

For example, you might define a set of runtime properties that points to test systems for flow development. When you move to production, you simply update the runtime property definitions in the file. This effectively updates every flow that uses those runtime properties without having to edit the flows.

When defining a runtime property, you can use a static value or an environment variable.

When calling the runtime property, you can use it as part of a larger property definition. For example, you can set a runtime property to the HOME environment variable, which would differ on different machines, and then call the runtime property as a base directory for a longer directory.

To implement runtime properties, perform the following steps:

Define runtime properties.
Use an expression in the flow to call a runtime property.

Step 1. Define runtime properties

You can define runtime properties in the Data Collector configuration properties, or in a separate runtime properties file.

Data Collector configuration properties

Use the following steps to define runtime properties in the Data Collector configuration properties:

On the Manage tab of your project, edit the StreamSets environment. Open the Advanced Configuration dialog box, then add the following Data Collector property as a key and value:
```
runtime.conf.location=embedded
```
To define the runtime properties, use either of the following formats to add key/value properties:
- To define a static value for a runtime property, use the following format:
```
runtime.conf_<property name>=<value>
```
  For example, the following runtime property defines a directory template for a Local FS target:
```
runtime.conf_LocalFSDirTemplate=/LocalFS/DirectoryTemplate
```
- To define an environment variable for a runtime property, use the following format:
```
runtime.conf_<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
runtime.conf_BaseDir=${env("HOME")}
```

Separate runtime properties file

Use the following steps to define runtime properties in a separate runtime properties file:

Create a text file and then use either of the following formats to define the runtime properties:
- To define a static value for a runtime property, use the following format:
```
<property name>=<value>
```
  For example, the following runtime property defines a directory template for a Local FS destination:
```
LocalFSDirTemplate=/LocalFS/DirectoryTemplate
```
- To define an environment variable for a runtime property, use the following format:
```
<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
BaseDir=${env("HOME")}
```
Add the file as an external resource for the engine, as described in Setting up external resources.
Save the changes to the StreamSets environment and restart all engine instances.

Step 2. Call the runtime property

Use the runtime:conf function to call a runtime property. You can use runtime properties to represent any stage or flow property that allows the use of the expression language.

To call a runtime property, use the following syntax:

${runtime:conf(<property name>)}

For example, to call the LocalFSDirTemplate runtime property to provide the directory template value for the Local FS target, enter the following syntax for the Directory Template property:

${runtime:conf('LocalFSDirTemplate')}

You can call a runtime property to represent a part of a property value. For example, you can call the RootDir runtime property and append the rest of the directory, as follows:

${runtime:conf('RootDir')}/logfiles