Engine log configuration

The Data Collector engine uses the Apache Log4j 2.x library to write log data.

The engine uses a default log configuration. You can change the log configuration by editing the StreamSets environment.

You might change the log configuration for the following reasons:

Modifying the log level

About this task

If the Data Collector engine log does not provide enough information for troubleshooting, you can modify the log level in the StreamSets environment.

By default, the engine logs most messages at the INFO level. You can configure the following log levels:
  • TRACE
  • DEBUG
  • INFO (default)
  • WARN
  • ERROR
  • FATAL

Procedure

  1. On the Manage tab of your project, edit the StreamSets environment.
  2. Expand the Advanced configuration section.
  3. In the Log4j2 properties section, click Add value twice.
  4. Enter the following keys and values:
    Key Value
    logger.l1.name com.streamsets
    logger.l1.level Log level to use, such as TRACE or DEBUG.
  5. Click Save.
  6. If the engine is running, restart the engine container for the changes to take effect.

Results

After troubleshooting, reset the log level to INFO to avoid having verbose log files.

Customizing the log configuration

About this task

For advanced use cases, you can further customize the log configuration by defining a custom appender. For information about defining a custom appender, see the Log4j 2.x documentation.

Important: You cannot override individual properties that are defined in a default appender. Instead, you must define a custom appender to change property values.
The Data Collector engine uses the following default log configuration that defines a null, stdout, and streamsets appender:
name = datacollectorLog

# /dev/null appender
appender.null.type = File
appender.null.name = null
appender.null.fileName = /dev/null
appender.null.layout.type = PatternLayout
appender.null.layout.pattern = -

# <stdout> appender
appender.stdout.type = Console
appender.stdout.name = stdout
appender.stdout.layout.type = PatternLayout
appender.stdout.layout.pattern = %d{DEFAULT} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] [stage:%X{s-stage}] %-5p %c{1} - %m%n
appender.stdout.filter.threshold.type = ThresholdFilter
appender.stdout.filter.threshold.level = ALL

# sdc.log appender
appender.streamsets.type = RollingFile
appender.streamsets.name = streamsets
appender.streamsets.fileName = ${sys:sdc.log.dir}/sdc.log
appender.streamsets.filePattern = ${sys:sdc.log.dir}/sdc-%i.log
appender.streamsets.layout.type = PatternLayout
appender.streamsets.layout.pattern = %d{DEFAULT} [user:%X{s-user}] [pipeline:%X{s-entity}] [runner:%X{s-runner}] [thread:%t] [stage:%X{s-stage}] %-5p %c{1} - %m%n
appender.streamsets.filter.threshold.type = ThresholdFilter
appender.streamsets.filter.threshold.level = ALL
appender.streamsets.policies.type = Policies
appender.streamsets.policies.size.type = SizeBasedTriggeringPolicy
appender.streamsets.policies.size.size=256MB
appender.streamsets.strategy.type = DefaultRolloverStrategy
appender.streamsets.strategy.fileIndex = min
appender.streamsets.strategy.max = 10

rootLogger.level = INFO
rootLogger.appenderRef.streamsets.ref = streamsets

logger.l1.name = com.streamsets
logger.l1.level = ${sdc.log4j.level}

logger.l2.name = org.eclipse.jetty
logger.l2.level = WARN

logger.l3.name = com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator
logger.l3.level = WARN

logger.l4.name = com.datastax.driver.core.QueryLogger.SLOW
logger.l4.level = DEBUG

Procedure

  1. On the Manage tab of your project, edit the StreamSets environment.
  2. Expand the Advanced configuration section.
  3. In the Log4j2 properties section, click Add value for each property that is defined in the custom appender.
  4. Enter the appropriate key and value for each custom appender property.
  5. Click Save.
  6. If the engine is running, restart the engine container for the changes to take effect.