Performance tuning

Use the parameters in the following sections to tune the performance of your MapReduce cluster.

C++ MapReduce framework
The C++ MapReduce framework replaces parts of the IBM® Spectrum Symphony MapReduce Java code with C++ code to achieve improved CPU and memory utilization. Within the IBM Spectrum Symphony MapReduce framework, a C++ implementation of the mapper and reducer is enabled, by default.
Data compression
Data compression offers two major benefits: it reduces the space needed to store files and it speeds up data transfer across the network, or to/from disk.
JVM options for map and reduce tasks
Within the MapReduce framework, a cluster or consumer administrator can configure standard Java™ options that are to be passed to the JVM upon invocation. You can specify any of the standard options to change JVM behavior, to enhance performance, or to support debugging separately for each map and reduce task.
Reduce tasks started based on Map tasks finished
Within the MapReduce framework in IBM Spectrum Symphony, you can specify the proportion of the total number of map tasks in a job that must be completed before any reduce tasks are scheduled.
Intermediate data locations
Within the MapReduce framework in IBM Spectrum Symphony, you can configure one or more locations for intermediate data that is generated by the map and reduce tasks.
Service instance sharing across jobs
You can share a service instance (SI) among multiple but similar MapReduce jobs to use data and state information cached in memory for better performance.
Configuring circular buffering for map tasks
A circular buffer can be used instead of the default dual buffers to temporarily save intermediate data from map tasks. Use the pmr.map.output.buffer.type parameter to specify the circular buffering approach.
Configuring dual buffers for map tasks
Either dual buffers or a circular buffer can be used to temporarily save intermediate data for map tasks. Dual buffers are the default. Use the pmr.ondemand.2nd.sort.mb parameter to specify whether the second map sort buffer is to be allocated for the job always or only when required. This parameter helps determine memory allocation for sorting files (specified as io.sort.mb) and the maximum heap size for a map task (specified as the -Xmx JVM option).
Hash-based aggregation for key sorting
Within the MapReduce framework, key sorting occurs during the spill and merge phase of a map task and during the fetch-merge and sort phase of a reduce task. Within the map phase, all <key, value> pairs are compared first by partition ID (integer), then by key (of the type specified by the user). Within the reduce phase, the reduce always and sometimes unnecessarily sorts data by key. Avoid this key sorting on mappers and reducers, especially inside partitions when a strict order between keys is not needed, by configuring a hash table-based aggregation. This configuration also improves the performance of certain types of queries.
Enabling fast byte array comparison
Within the MapReduce framework, the shuffle phase of a job when the input to every reducer is sorted by key can consume CPU resources, which impact job performance. To improve the performance of the sorting process, enable fast byte array comparison to decide on an ordering between two keys by looking at the byte streams and without parsing all the data contained therein. By default, fast byte comparison is disabled.
Concurrent merge
During the reduce phase of a MapReduce job, a reduce task's copiers fetch all map outputs for its particular partition in parallel. The map outputs are copied and decompressed to MRSS memory if the output is small; otherwise, the output is written directly to disk. When the in-memory buffer reaches a threshold size or reaches a threshold number of map outputs, the map output is merged and compressed to disk. As the copies accumulate on disk, a merger thread decompresses and merges the output. It then compresses and writes the output to larger sorted files, during what is known as the intermediate merge. When all the map outputs are copied, the reduce task decompresses and merges the map outputs, maintaining their sort ordering, during the final merge. The results are then directly fed into the application reduce function.

Parent topic: MapReduce configuration