DCC logging

Log files are used to record events that occur on specific demands, tasks, errors, or exceptions. Logs files are useful in debugging and resolving the issues.

Data Collection Component creates separate log file based on the jobs and job name. The job thread name runtime is appended into the log file name.

Data Collection Component uses an updated version of the log4j library. The log4j2 has all the logging features of its predecessor and builds on that foundation with some significant improvements, especially in the area of performance.

Customizing the log4j2 configuration

A custom log4j2 configuration can be created by either using a program or a configuration file. The library supports configuration files that are written in XML, JSON, YAML, and .properties formats. Data Collection Component uses XML format to configure its properties. The log4j2.xml file is the configurable file, which is used for the logging and tracing techniques. The location of the configuration file is Install_Dir/server/conf/dcc/log4j2.xml. This file is loaded at the server startup and is automatically reloaded if any change is detected in the file. The log4j2.xml file is checked for changes every 10 seconds. This configuration file can be set to a different tracing technique, which is known as the log level. The following log levels can be used:

  • off
  • fatal
  • error
  • warning
  • informational
  • debug
  • trace
  • all
The default value is WARN.
The following example shows a possible configuration of the log4j file:

<?xml version="1.0" encoding="UTF-8"?>
    <Configuration status="WARN">
                <Console name="Console" target="SYSTEM_OUT">
                         <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
             <Root level="INFO">
                   <AppenderRef ref="Console"/>

Logs directory

The logs directory is where all Data Collection Component logs are stored, and its location can vary according to the installation process.

The default application server uses the following locations:
  • Install_Dir/server/logs/dcc_etl_*********.log
  • Install_Dir/server/logs/dcc_etl_*********.log
The location of the log files is determined by the dir property name in the log4j2.xml file. The following default setting is used for the xx.log file:
<Property name="dir">logs</Property>

Configuring appenders

In the log4j2 architecture, an appender is responsible for sending log messages to a certain output destination.

The following appenders are used in Data Collection Component:
  • ConsoleAppender – logs messages to the System console.
  • FileAppender – writes log messages to a file.
  • RollingFileAppender – writes the messages to a rolling log file.

Configuring loggers

In addition to the Root logger, the Logger elements can be defined with different log levels, appenders, or filters. Each Logger requires a name that can be used to reference it later.

           <Logger name="RollingFileLogger">
                <AppenderRef ref="RollingFileAppender" />

The log file size

The triggering policy determines when the log file is rolled, which means that a new file is created, and the rollover strategy determines how the file is rolled. By default, the size of the log file is set to 10 MB, and the number of backup files is set to 5 in the log4j2.xml file. This file is in the Install_dir/server/conf/dcc directory. You can specify the maximum number of the archived log files on the same day. After the maximum value is reached, older archives are deleted on subsequent rollovers.

To do a size-based rollover, update the numeric value of the following lines:

  • <SizeBasedTriggeringPolicy size="10MB"/>
  • <DefaultRolloverStrategy max="5" />

Custom log levels

Custom log levels can be defined in the code or in the configuration. To define a custom log level in the configuration, update the root log level to a specific level. The configuration logs all the specific level-related logs in the file.

To update the log level to trace, change the Root level property in the following ways:

  • Find the section:
    <Root level="INFO">
              <AppenderRef ref="file" />
  • Update the section to:
    <Root level="TRACE">
            <AppenderRef ref="file" />

Increasing the size of the log file

The size can be specified in bytes by using the suffix KB, MB, or GB. The default value is 10 MB. After SizeBasedTriggeringPolicy reaches the size that you specify, a rollover occurs. When the SizeBasedTriggeringPolicy is triggered, the log file is compressed into the .gz format file.

Archive the log file according to the following patterns:

  • <SizeBasedTriggeringPolicy size="10MB"/>
    default size.
  • <SizeBasedTriggeringPolicy size="100MB"/>
    can be increased.

Extended use of appender to give dynamic file name

Data Collection Component extends the additional features of RollingFileAppender to programmatically rename the log file name according to the job name. RollingFileAppender provides a list of methods to set properties dynamically.
RollingFileAppender appender = new RollingFileAppender();
appender.setFile(fileName + "_" + dateFormat.format(date) + ".log");