Data Collector configuration

You can configure Data Collector configuration properties, such as the host name and port number, when you configure the StreamSets environment.

You can protect sensitive data in Data Collector configuration properties by storing the data in an external location and then using functions provided with the IBM StreamSets expression language to retrieve the data. You can also reference information in an environment variable.

You can define runtime properties in Data Collector configuration properties or in a separate file. For more information, see Runtime properties.

Learn more

Blocklist and allowlist for stage libraries

By default, almost all installed stage libraries are available for use in Data Collector. You can use blocklist and allowlist properties to limit the stage libraries that can be used.

To limit the stage libraries created by IBM StreamSets, use one of the following properties:

system.stagelibs.allowlist
system.stagelibs.blocklist

To limit stage libraries created by other parties, use one of the following properties:

user.stagelibs.allowlist
user.stagelibs.blocklist

Warning: Use only the allowlist or blocklist for each set of libraries. Using both can cause Data Collector to fail to start.

Configure Data Collector engine properties

About this task

You can customize the Data Collector engine by configuring the StreamSets environment. On the Manage tab of your project, edit the environment and then expand the Advanced configuration section.

For each property you want to configure, in the Engine properties section, click Add value. Enter the property name under Key and the property value under Value.

Important: Instead of entering sensitive data such as passwords in clear text in the configuration properties, you can by storing the data in an external location and then using functions to retrieve the data.

You can configure the following general configuration properties:

General Property	Description
sdc.base.http.url	Data Collector URL. Default is `http://<hostname>:<http.port>` where `<hostname>` is the value defined in the http.bindHost property. If the host name is not defined in http.bindHost, Data Collector runs the following command to determine the host name: `hostname -f`
sdi.engine.token.renewal.interval	Time between token renewals in seconds. IBM tokens have a life of 60 minutes. Default is 2700.
sdi.jobrunner.force.stop.timeout	Maximum time in milliseconds a flow can be in a Stopping state before Data Collector forces the flow to stop. Default is 120000.
sdi.jobrunner.runner.force.quit.timeout	Maximum time in milliseconds Data Collector waits for the flow runner to stop before manually changing the flow state. Default is 10000.
sdi.jobrunner.parallel.actions.timeout	Maximum time in seconds to wait to execute all actions in parallel. After the timeout interval is reached, Data Collector waits for the amount of time configured in the sdi.jobrunner.syncup.interval property before querying the IBM watsonx platform for new actions. Default is 30.
sdi.jobrunner.parallel.actions.concurrency	Maximum number of concurrent actions. Default is 20.
sdi.jobrunner.parallel.actions.backlog.size	Maximum number of actions in the backlog. Default is 1000.
sdi.flowstore.gateway.timeout	Maximum time in milliseconds Data Collector waits to receive flow configuration from the IBM watsonx platform. Default is 15000.
sdi.jobrunner.gateway.timeout	Maximum time in milliseconds Data Collector waits to receive actions from the IBM watsonx platform. Default is 10000.
sdi.jobrunner.syncup.interval	Number of seconds between syncs with the IBM watsonx platform. Default is 5.
http.bindHost	Host name or IP address that Data Collector binds to. You might want to configure a specific host or IP address when the machine that Data Collector is installed on has multiple network cards. Default is 0.0.0.0, which means that Data Collector can bind to any host or IP address..
http.port	Port number to use for Data Collector. If you configure both http.port and https.port with different values, the HTTP port bounces to the HTTPS port. Default is 18630.
https.port	Secure port number for Data Collector. For example, 18636. If you use both port properties, the HTTP port bounces to the HTTPS port. Default is 18630. If you configure both http.port and https.port with different values, the HTTP port bounces to the HTTPS port. For more information, see Enabling HTTPS host verification.
http2.enable	Enables support of the HTTP/2 protocol for the API. To enable HTTP/2, set this property to `true` and configure the https.port property, above. Do not use with clients that do not support application layer protocol negotiation (ALPN). Default is `false`.
http.enable.forwarded.requests	Enables handling X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Port HTTP request headers issued by a reverse proxy such as HAProxy, ELB, or NGINX. Set to `true` when hosting Data Collector behind a reverse proxy or load balancer. Default is `false`.
https.keystore.path	Keystore path and file name used by Data Collector. Enter an absolute path or a path relative the Data Collector resources directory. Note: Default is `keystore.jks` in the Data Collector configuration directory which provides a self-signed certificate that you can use. However, it is best practice to generate a certificate signed by a trusted CA, as described in Enabling HTTPS host verification.
https.keystore.password	Password to the Data Collector keystore file. Default uses the `file` function to retrieve the password from keystore-password.txt in the Data Collector configuration directory.
https.require.hsts	Requires Data Collector to include the HTTP Strict Transport Security (HSTS) response header. Set to `true` when Data Collector uses HTTPS to enable HSTS. Default is `false`.
http.access.control.allow.origin	List of domains allowed to access the Data Collector REST API for cross-origin resource sharing (CORS). To restrict access to specific domains, enter a comma-separated list as follows: `http://www.mysite.com, http://www.myothersite.com` Default is the asterisk wildcard (*) which means that any domain can access the Data Collector REST API.
http.access.control.allow.headers	List of HTTP headers allowed during a cross-domain request.
http.access.control.exposed.headers	List of HTTP headers exposed as part of the cross-domain response.
http.access.control.allow.methods	List of HTTP methods that can be called during a cross-domain request.
kerberos.client.enabled	Enables Kerberos authentication for Data Collector. Must be enabled to allow non-Kafka stages to use Kerberos to access external systems.
kerberos.client.principal	Kerberos principal to use. Enter a service principal.
kerberos.client.keytab	Location of the Kerberos keytab file that contains the credentials for the Kerberos principal. Use a fully-qualified directory or a directory relative to the Data Collector configuration directory.
production.maxBatchSize	Maximum number of records included in a batch when the flow runs. Default is 50000.
parser.limit	Maximum parser buffer size that sources can use to process data. Limits the size of the data that can be parsed and converted to a record. By default, the parser buffer size is 1048576 bytes. To increase the size, configure this property. For more information about how this property affects record sizes, see Maximum record size.
production.maxErrorRecordsPerStage	Maximum number of error records to save in memory for each stage to display in Monitor mode. When the limit is reached, older error records are discarded. Default is 100.
production.maxPipelineErrors	Maximum number of flow errors to save in memory to display in monitor mode. When the limit is reached, older errors are discarded. Default is 100.
max.logtail.concurrent.requests	Maximum number of external processes allowed to access the Data Collector log file at the same time through REST API calls. Default is 5.
max.webSockets.concurrent.requests	Maximum number of WebSocket calls allowed.

You can configure the following properties for sending email:

Email Property	Description
mail.transport.protocol	Use smtp or smtps. Default is `smtp`.
mail.smtp.host	SMTP host name. Default is `localhost`.
mail.smtp.port	SMTP port number. Default is 25.
mail.smtp.auth	Whether the SMTP host uses authentication. Use `true` or `false`. Default is `false`.
mail.smtp.starttls.enable	Whether the SMTP host uses STARTTLS encryption. Use `true` or `false`. Default is `false`.
mail.smtps.host	SMTPS host name. Default is `localhost`.
mail.smtps.port	SMTPS port number. Default is 25.
mail.smtps.auth	Whether the SMTPS host uses authentication. Use `true` or `false`. Default is `false`.
xmail.username	User name for the email account to send email.
xmail.password	Password for the email account. To protect the password, store the password in an external location and then use a function to retrieve the password. Default uses the `file` function to retrieve the password from email-password.txt in the Data Collector configuration directory, <installation_dir>/etc.
xmail.from.address	Email address to use to send email.

You can configure the following advanced properties:

Advanced Property	Description
runtime.conf.location	Location of runtime properties. Use to declare where runtime properties are defined: `embedded` - Runtime properties are defined in the Data Collector configuration properties.

You can configure the following stage-specific properties:

Stage-Specific Properties	Description
stage.conf_hadoop.always.impersonate.current.user	Ensures that Hadoop-related stages use the currently logged in Data Collector user to perform tasks, such as writing data, in Hadoop systems. With this property enabled, Data Collector prevents configuring an alternate user in Hadoop-related stages. To use this property, set it to `true`.
	Converts the user name to lowercase before passing it to Hadoop. Use to lowercase user names from case insensitive systems, such as a case-insensitive LDAP installation, before passing the user names to Hadoop systems. To use this property, set it to `true`.
stage.conf_com.streamsets.pipeline.stage.jdbc.drivers.load	Lists JDBC drivers that Data Collector automatically loads for all flows. To use this property, set it to a comma-separated list of JDBC drivers.
stage.conf_com.streamsets.pipeline.lib.jdbc.disableSSL	Enables Data Collector to attempt to disable SSL for all JDBC connections. Many newer JDBC systems enable SSL by default. When you have JDBC flows that do not use SSL, you can use this property to handle JDBC systems with SSL enabled. However, some JDBC vendors do not allow disabling SSL. To use this property, set it to `true`.
stage.conf_kafka.keytab.location	Storage location for Kerberos keytabs that are specified in Kafka stages. Keytabs are stored only for the duration of the flow run. Generally, you should not need to change this property.
stage.conf_com.streamsets.pipeline.stage.origin.jdbc.cdc. oracle.addrecordstoqueue	Enables the Oracle CDC Client source to reduce memory usage when the source is configured to buffer data locally, in memory. This property is enabled by default. Do not disable this property unless recommended by customer support.
stage.conf_com.streamsets.pipeline.stage.origin.jdbc.cdc. oracle.monitorbuffersize	Enables Data Collector to report memory consumption when the Oracle CDC Client source uses local buffers. Reporting reduces flow performance, so enable the property only as a temporary troubleshooting measure. This property is disabled by default.
stage.conf_com.streamsets.pipeline.stage.executor.shell. shell	Defines the relative or absolute path to the command line interpreter to use to execute scripts, such as `/bin/bash`. Default is `sh`. Used by Shell executors.
stage.conf_com.streamsets.pipeline.stage.executor. shell.sudo	Defines the relative or absolute path to the sudo to use when executing scripts. Default is `sudo`. Used by Shell executors.

The Data Collector configuration properties include flight properties which you can use to configure IBM Connectivity Service properties. IBM Connectivity Service must be running to use stages included in the IBM Connectivity Service stage library, streamsets-datacollector-ibm-connectivity-service-lib.

You can configure the following IBM Connectivity Service properties:

IBM Connectivity Service Properties Description

flight.enable

IBM Connectivity Service Properties	Description
flight.enable	Determines whether Data Collector starts IBM Connectivity Service: auto - Data Collector starts IBM Connectivity Service only if the IBM Connectivity Service stage library is installed. true - Data Collector starts IBM Connectivity Service. false - Data Collector does not start IBM Connectivity Service.
flight.port	Port IBM Connectivity Service runs on. Default is 9443.
flight.java_opts	Optional string appended to the JAVA_OPTS environment variable when Data Collector starts IBM Connectivity Service. You can use any of the following strings to limit the amount of memory IBM Connectivity Service uses: `-XX:MaxRAM=[value]` - Defines the total amount of RAM IBM Connectivity Service can use. `-XX:InitialRAMPercentage=[percent]` - The initial percentage of RAM the IBM Connectivity Service JVM can use for the Java heap. Regardless of the configured percentage, the JVM will never use more than maximum amount of RAM defined in `-XX:MaxRAM`. Default is 1.5625 percent. -`XX:MaxRAMPercentage=[percent]` - The maximum amount of RAM the IBM Connectivity Service JVM can use for a Java heap of 200MB or larger. Regardless of the configured percentage, the JVM will never use more than maximum amount of RAM defined in `-XX:MaxRAM`. Default is 25 percent. `-XX:MinRAMPercentage=[percent]` - The maximum amount of RAM the IBM Connectivity Service JVM can use for a Java heap of less than 200 MB. Regardless of the configured percentage, the JVM will never use more than maximum amount of RAM defined in `-XX:MaxRAM`. Default is 50 percent.

Determines whether Data Collector starts IBM Connectivity Service:

auto - Data Collector starts IBM Connectivity Service only if the IBM Connectivity Service stage library is installed.
true - Data Collector starts IBM Connectivity Service.
false - Data Collector does not start IBM Connectivity Service.

flight.port

Port IBM Connectivity Service runs on.

Default is 9443.

flight.java_opts

Optional string appended to the JAVA_OPTS environment variable when Data Collector starts IBM Connectivity Service.

You can use any of the following strings to limit the amount of memory IBM Connectivity Service uses:

-XX:MaxRAM=[value] - Defines the total amount of RAM IBM Connectivity Service can use.
-XX:InitialRAMPercentage=[percent] - The initial percentage of RAM the IBM Connectivity Service JVM can use for the Java heap. Regardless of the configured percentage, the JVM will never use more than maximum amount of RAM defined in -XX:MaxRAM. Default is 1.5625 percent.
-XX:MaxRAMPercentage=[percent] - The maximum amount of RAM the IBM Connectivity Service JVM can use for a Java heap of 200MB or larger. Regardless of the configured percentage, the JVM will never use more than maximum amount of RAM defined in -XX:MaxRAM. Default is 25 percent.
-XX:MinRAMPercentage=[percent] - The maximum amount of RAM the IBM Connectivity Service JVM can use for a Java heap of less than 200 MB. Regardless of the configured percentage, the JVM will never use more than maximum amount of RAM defined in -XX:MaxRAM. Default is 50 percent.

You can configure the following miscellaneous properties:

Miscellaneous Property	Description
max.stage.private.classloaders	Maximum number of stage libraries in an environment. Default is 50.
runner.thread.pool.size	Pre-multiplier size of the thread pool. One running flow requires five threads, and flows share threads in the pool. To calculate the approximate runner thread pool size, multiply the number of running flows by 2.2. Increasing this value does not increase the parallelization of an individual flow. Default is 50, which is sufficient to run approximately 22 flows at the same time.
pipeline.max.runners.count	Maximum number of flow runners to use for a multithreaded flow. Default is 50.
package.manager.repository.links	Enables specifying alternate locations for the Package Manager repositories. Use this property to install non-IBM StreamSets stage libraries or to install stage libraries from local or alternate repositories. To use alternate Package Manager repositories, specify a comma-separated list of URLs.

You can optionally add stage libraries to the following properties to limit the stage libraries Data Collector uses.

Blocklist / Allowlist Property	Description
system.stagelibs.allowlist system.stagelibs.blocklist	Use one list to limit the IBM StreamSets stage libraries that can be used in Data Collector. Do not use both.
user.stagelibs.allowlist user.stagelibs.blocklist	Use one list to limit the third-party stage libraries that can be used in Data Collector. Do not use both.

You can configure the following classpath validation properties:

Classpath Validation Property Description

stagelibs.classpath.validation.enable

Classpath Validation Property	Description
stagelibs.classpath.validation.enable	Disables classpath validation when necessary. By default, Data Collector performs classpath validation each time it starts. It writes the results to the Data Collector log. Though generally unnecessary, you can disable classpath validation bysetting it to `false`.
stagelibs.classpath.validation.terminate	Prevents Data Collector from starting when it discovers an invalid classpath. To use enable this behavior, set this property to to `true`.

Disables classpath validation when necessary.

By default, Data Collector performs classpath validation each time it starts. It writes the results to the Data Collector log.

Though generally unnecessary, you can disable classpath validation bysetting it to false.

stagelibs.classpath.validation.terminate

Prevents Data Collector from starting when it discovers an invalid classpath.

To use enable this behavior, set this property to to true.

You can configure the following Health Inspector property:

Health Inspector Property	Description
health_inspector.network.host	Host name that the Data Collector Health Inspector uses for the `ping` and `traceroute` commands.

You can configure the following property that specifies additional configuration properties to include in the Data Collector configuration:

Additional Files Property	Description
config.includes	Additional configuration properties to include in the Data Collector configuration. You can enter multiple file names separated by commas. The files are loaded into the Data Collector configuration in the listed order. If the same configuration property is defined in multiple files, the value defined in the last loaded file takes precedence. By default, credential store and security policy properties are included in the Data Collector advanced configuration properties.

Additional Files Property

Description

config.includes

Additional configuration properties to include in the Data Collector configuration.

You can enter multiple file names separated by commas. The files are loaded into the Data Collector configuration in the listed order. If the same configuration property is defined in multiple files, the value defined in the last loaded file takes precedence.

By default, credential store and security policy properties are included in the Data Collector advanced configuration properties.

The Data Collector configuration properties include record sampling properties that indicate the size of the sample set chosen from a total population of records. Data Collector uses the sampling properties when you run a flow that writes to a target system using the SDC Record data format and then run another flow that reads from that same system using the SDC Record data format. Data Collector uses record sampling to calculate the time that a record stays in the intermediate target.

By default, Data Collector uses 1 out of 10,000 records for sampling. If you modify the sampling size, simplify the fraction for better performance. For example, configure the sampling size as 1/40 records instead of 250/10000 records. The following properties specify the sampling size:

Record Sampling Property	Description
sdc.record.sampling.sample.size	Size of the sample set. Default is 1.
sdc.record.sampling.population.size	Size of the total number of records. Default is 10,000.

The Data Collector configuration properties include properties that define how Data Collector caches flow states. Data Collector can cache the state of flows for faster retrieval of those states. If Data Collector does not cache flow states, it must retrieve flow states from the flow data files stored in the $SDC_DATA directory. You can configure the following properties that specify how Data Collector caches flow states:

Flow State Cache Property	Description
store.pipeline.state.cache.maximum.size	Maximum number of flow states that Data Collector caches. When the maximum number is reached, Data Collector evicts the oldest states from the cache. Default is 100.
store.pipeline.state.cache.expire.after.access	Amount of time in minutes that a flow state can remain in the cache after the entry's creation, the most recent replacement of its value, or its last access. Default is 10 minutes.