IC5Notice: We have upgraded developerWorks Community to the latest version of IBM Connections. For more information, read our upgrade FAQ.
  • No replies
Nikhil Sood
Nikhil Sood
1 Post

Pinned topic Hadoop Map job failing on IBM Power 6 Linux Node

‏2013-07-10T11:08:53Z |



I have a POWER6 Linux Node setup as a slave in a Hadoop 1.0.4 cluster (Red Hat 4.1.2-48) Master node is RHEL node on Wintel virtual partition.

On Master Sun JDK 1.6.43 is installed, and on Power slave node, IBM JDK 1.6.0 is installed.

When I submit a pig script to be run on master node, it submits MAP jobs on all slave nodes and the ones on the POWER6 node fail with the following error:

10 10:52:36,610 WARN org.apache.hadoop.mapred.Child: Error running child.java.lang.NullPointerException at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:102) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1407) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(AccessController.java:310) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)

When I see the log up above, I see a warning as below: 2013-07-10 10:52:21,698 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2013-07-10 10:52:22,690 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded

I assume that native libraries for HADOOP are unable to be loaded.

This problem doesnt happen when I install SUN JDK on other nodes, but this power node has IBM JDK.

The path of Java on POWER6 Slave node is /opt/ibm/java-ppc64-60/bin/java and Hadoop path is /usr/local/hadoop/bin/hadoop

I read an article https://issues.apache.org/jira/browse/DAEMON-249 and created a commons-daemon-1.0.2-bin-linux-ppc64.tar.gz file but I dont know what to do with this tarball and how to implement the changes.

Being new to the Linux and Hadoop community, I am self learning and doing these tasks and am stuck as I cant add a Power node to my cluster .

A detailed and quick help for fixing this problem will be highly appreciated.