Logging

The map/reduce framework incorporates log4j loggers. All logs are saved in the $MPAREDUCE_HOME/jobs/<job_dir>/logs directory. Logs are divided into the following groups:

Table 1. Available logs in the map/reduce framework
Log	Description
Main log
main.log	Contains basic information about the job, such as validating args, validating job, job completion status, number of counters, deploy directory, number of mapper/reducer/combiner tasks.
Shaper's logs
mapper_shaper.log	Contains success notification or error messages regarding running the shaper during the map phase.
combiner_shaper.log	Contains success notification or error messages regarding running the shaper during the combiner phase.
reducer_shaper.log	Contains success notification or error messages regarding running the shaper during the reducer phase.
Tasks' logs
mapper<id>.log	Each mapper creates one log file containing mapperspecific information (for example, number of records read by the mapper, number of bad records, number of output records).
combiner<id>.log	An optional log file that is created only if combiners were used. These files contain basic combiner-specific information such as number of combine input records and number of combine output records.
reducer<id>.log	An optional log file that is created only if reducers were used. These files contain basic reducer-specific information such as number of records read by the reducer and number of output records.

You can also use your own loggers to provide more detailed information in the log files. To do this, you must acquire a Logger instance in your source code (see the following example, with the Logger instance related lines marked in bold).

The following is an example using a user-defined logger:

import org.apache.log4j.Logger;
import …
public class WordCount extends Configured implements Tool {
static Logger log = Logger.getLogger(WordCount.class);
...
public static class Map extends Mapper<LongWritable, Text, Text,
IntWritable> {
...
@Override
public void map(LongWritable key, Text value, Context
context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
log.info(“word = ” + word);
context.write(word, one);
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text,
IntWritable> {
...
@Override
public void reduce(Text key, Iterable<IntWritable> values,
Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
totalSum.set(sum);
log.info(“sum = ” + sum);
context.write(key, totalSum);
}
}
public int run(String[] args) throws Exception {
Job job = new Job(getConf(), getClass().getSimpleName());
...
}
public static void main(String[] args) throws Exception {
int ret = ToolRunner.run(new WordCount(), args);
System.exit(ret);
}
}