Data movement log files

For troubleshooting purposes, you can configure logging for the data movement feature. You can also review mapper logs when import or export fails.

About this task

When the data import or export operation fails, you might refer to the following log files for mappers:
  • on Hadoop: /fastDataMovement/var/log/mapper.
  • on Db2 Warehouse: by default all logs are displayed on stdout.
You can also use the following procedure to save additional log file when running data movement.

Procedure

  • On Hadoop:
    1. Create a log4j.properties file in the directory where you start the import and export operations.
    2. Paste the following code in the file: 
      # initialize root logger with level ERROR for stdout and fout
log4j.rootLogger=DEBUG,file
log4j.logger.com.ibm.nz=DEBUG,stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.Threshold=INFO
log4j.appender.stdout.layout.ConversionPattern=%d %p [Thread-%t] %c{2}: %m%n

# add a FileAppender to the logger file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.File=fq.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d %p [Thread-%t] %c{2}: %m%n
      The logs will be produced in a fq.log file in the same directory.
  • On Db2 Warehouse:

    If you want to change the default behaviour for FDM logger, create log4j.properties file in the same directory that you use to execute the fdm_iias command. The following example shows how to send all logs to stdout and to the file as well, using log4j.properties file created in bluadmin's home directory:

    [bluadmin@nc143206 - Db2wh ~]$ cd ~

[bluadmin@nc143206 - Db2wh ~]$ vi log4j.properties
# Root logger option

log4j.rootLogger=DEBUG, file, stdout


# Direct log messages to a log file

log4j.appender.file=org.apache.log4j.RollingFileAppender

log4j.appender.file.File=/tmp/fdm.log

log4j.appender.file.MaxFileSize=10MB

log4j.appender.file.MaxBackupIndex=10

log4j.appender.file.layout=org.apache.log4j.PatternLayout

log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n


# Direct log messages to stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.Target=System.out

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

[bluadmin@nc143206 - Db2wh ~]$ fdm_iias -hadoop_jars ~/hadoop_libs/horton/fqdm/ -conf ~/example/fq-export-remote-conf.xml

Fast Data Movement version: 1.0.3.0 [Build 190509-4]

2019-05-23 14:58:47 INFO  LogConfigurator:90 - Using: log4j.properties

2019-05-23 14:58:47 INFO  CommandArguments:100 - PDA jar: /opt/ibm/db2/V11.1.0.0/java/db2jcc4.jar

2019-05-23 14:58:47 DEBUG CommandArguments:117 - Add -libjars

2019-05-23 14:58:47 DEBUG CommandArguments:134 - Args[libjars]: /opt/ibm/db2/V11.1.0.0/java/db2jcc4.jar

2019-05-23 14:58:47 DEBUG NzTransfer:217 - Libjars: /opt/ibm/db2/V11.1.0.0/java/db2jcc4.jar

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/mnt/blumeta0/home/bluadmin/hadoop_libs/horton/fqdm/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/mnt/blumeta0/home/bluadmin/hadoop_libs/horton/fqdm/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

... further output omitted ...


[bluadmin@nc143206 - Db2wh ~]$ ls -ll /tmp/fdm.log

-rw-rw-r--. 1 bluadmin bluadmin 39538 May 23 14:58 /tmp/fdm.log