• No replies
Nikhil Sood
Nikhil Sood
1 Post

Pinned topic Hadoop Map job failing on IBM Power 6 Linux Node

‏2013-07-10T11:08:53Z |



I have a POWER6 Linux Node setup as a slave in a Hadoop 1.0.4 cluster (Red Hat 4.1.2-48) Master node is RHEL node on Wintel virtual partition.

On Master Sun JDK 1.6.43 is installed, and on Power slave node, IBM JDK 1.6.0 is installed.

When I submit a pig script to be run on master node, it submits MAP jobs on all slave nodes and the ones on the POWER6 node fail with the following error:

10 10:52:36,610 WARN org.apache.hadoop.mapred.Child: Error running at org.apache.hadoop.mapred.IFile$Writer.( at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill( at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush( at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close( at org.apache.hadoop.mapred.MapTask.runNewMapper( at at org.apache.hadoop.mapred.Child$ at at at

When I see the log up above, I see a warning as below: 2013-07-10 10:52:21,698 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2013-07-10 10:52:22,690 WARN Snappy native library not loaded

I assume that native libraries for HADOOP are unable to be loaded.

This problem doesnt happen when I install SUN JDK on other nodes, but this power node has IBM JDK.

The path of Java on POWER6 Slave node is /opt/ibm/java-ppc64-60/bin/java and Hadoop path is /usr/local/hadoop/bin/hadoop

I read an article and created a commons-daemon-1.0.2-bin-linux-ppc64.tar.gz file but I dont know what to do with this tarball and how to implement the changes.

Being new to the Linux and Hadoop community, I am self learning and doing these tasks and am stuck as I cant add a Power node to my cluster .

A detailed and quick help for fixing this problem will be highly appreciated.