New results for Hadoop performance testing
Anirban C 110000R45D Comments (3) Visits (6109)
By: Anirban Chatterjee.
The team has not been standing still, however. With the launch in February of our new 7R2s that included enhanced POWER7+ processors, the team has pushed the envelope even further on these systems and, with a similarly sized cluster, is now able to sort a terabyte of data in less than 6.7 minutes.
The IBM China Research Lab reached this milestone using a 10-node cluster running RHEL 6.2 and Hadoop 1.1.3, managed with IBM Platform Symphony. The cluster comprised of one master control node and nine compute nodes. At 16 cores per compute node, this amounts to a sorting rate of 1.04 GB/min/core. (By comparison, a recent benchmark using an 18-node Cloudera Hadoop cluster of HP ProLiant Gen8 DL380 systems achieved a sorting rate of 0.57 GB/min/core.*)
We’ll have more information on the details of the testing environment coming soon, but proof points like this show the ability of Power Systems and Platform Symphony to provide high performance data analytics platforms at a reasonable cost. IBM solutions can provide rapid results to big data challenges, often in half the time as other solutions.