Building native Hadoop libraries on Linux on POWER

You may require native Hadoop libraries for certain Hadoop features, particularly if you are using the Gzip compression format for data compression on Linux®on POWER systems. Because IBM® Spectrum Symphony does not ship with the native Hadoop libraries, you must build the native libraries for Linux on POWER and then set up IBM Spectrum Symphony to use them.

About this task

Follow these steps to build the native Hadoop libraries on Linuxon POWER and include the libraries in the IBM Spectrum Symphony classpath.

Procedure

Build the Hadoop Gzip native library (libhadoop.so) for the Linuxon POWER platform, as described in Hadoop documentation: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html

Note: To build your Hadoop version on PowerLinux, there are particular considerations, most of which relate to the use of IBM Java™.

Assuming that you built the native libraries under the $HADOOP_HOME directory, you will find the native library at the following location: native/os_type/lib/libhadoop.so, where os_type specifies the operating system, such as Linux-ppc64-64.
Copy the native/os_type/lib folder and all its contents to $HADOOP_HOME/lib/native/ and rename the lib folder as os_type (for example, Linux-ppc64-64).
To enable the MapReduce GzipCodec Java class to load the native library at run time, include the path to the native library to the IBM Spectrum Symphony classpath.
1. Open the pmr-env.sh configuration file at $PMR_HOME/conf/.
2. Look for JAVA_LIBRARY_PATH and replace:
```
export JAVA_LIBRARY_PATH=@HADOOP_HOME@/lib/native/Linux-amd64-64/:@HADOOP_HOME@/lib/native/Linux-i386-32/
```
  with:
```
export JAVA_LIBRARY_PATH=@HADOOP_HOME@/lib/native/Linux-ppc64-64
```
3. Save the file.