Topic
1 reply Latest Post - ‏2013-12-12T13:15:53Z by ECJ3_Christof_Buechi
yong8964
yong8964
1 Post
ACCEPTED ANSWER

Pinned topic Failed to enable snappy

‏2013-10-04T20:48:28Z |

We are using BigInsight 2.1 Enterprise version. 

I am running it on Suse Linux x86_64. I am trying to enable the snappy in the hadoop MR job.

First, it will give me the following error:

Exception in thread "main" java.lang.RuntimeException: native snappy library not available

Then I installed the snappy 1.0.5, which is the integrated version for my SUSE.

After that, I also copied libsnappy.so file to $BIGINSIGHT_HOME/IHC/lib/native/Linux-amd64-64/

Then when I tested my MR job, this time it gave back different error:

2013-10-04 16:32:01,539 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-10-04 16:32:01,869 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-10-04 16:32:01,877 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@b990b99
2013-10-04 16:32:02,017 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2013-10-04 16:32:02,024 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 256
2013-10-04 16:32:02,095 INFO org.apache.hadoop.mapred.MapTask: data buffer = 204010960/255013696
2013-10-04 16:32:02,095 INFO org.apache.hadoop.mapred.MapTask: record buffer = 671088/838860
2013-10-04 16:32:02,293 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
2013-10-04 16:32:02,293 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
2013-10-04 16:32:23,909 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2013-10-04 16:32:24,342 WARN org.apache.hadoop.io.compress.snappy.SnappyCompressor: java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.initIDs()V
2013-10-04 16:32:24,342 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
2013-10-04 16:32:24,403 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-10-04 16:32:24,418 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-10-04 16:32:24,418 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName yzhang for UID 1000 from the native implementation
2013-10-04 16:32:24,419 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I
        at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229)
        at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:141)
        at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:135)
        at org.apache.hadoop.mapred.IFile$Writer.close(IFile.java:135)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1450)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(AccessController.java:310)
        at javax.security.auth.Subject.doAs(Subject.java:573)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)

It looks like the hadoop knew that the native snappy is there, as shown  in the following log:

2013-10-04 16:32:02,293 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
2013-10-04 16:32:02,293 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded

But why it gave the above error?