Concurrent merge

During the reduce phase of a MapReduce job, a reduce task's copiers fetch all map outputs for its particular partition in parallel. The map outputs are copied and decompressed to MRSS memory if the output is small; otherwise, the output is written directly to disk. When the in-memory buffer reaches a threshold size or reaches a threshold number of map outputs, the map output is merged and compressed to disk. As the copies accumulate on disk, a merger thread decompresses and merges the output. It then compresses and writes the output to larger sorted files, during what is known as the intermediate merge. When all the map outputs are copied, the reduce task decompresses and merges the map outputs, maintaining their sort ordering, during the final merge. The results are then directly fed into the application reduce function.

When you configure concurrent file-to-file merge, instead of a single thread, multiple merger threads decompress and merge the map output, enabling each merge thread to work on a subset of segments or files to speed up the file-to-file merge. This is done both during the intermediate and final merges.