• 1 reply
  • Latest Post - ‏2013-12-12T13:15:53Z by ECJ3_Christof_Buechi
1 Post

Pinned topic Failed to enable snappy

‏2013-10-04T20:48:28Z |

We are using BigInsight 2.1 Enterprise version. 

I am running it on Suse Linux x86_64. I am trying to enable the snappy in the hadoop MR job.

First, it will give me the following error:

Exception in thread "main" java.lang.RuntimeException: native snappy library not available

Then I installed the snappy 1.0.5, which is the integrated version for my SUSE.

After that, I also copied file to $BIGINSIGHT_HOME/IHC/lib/native/Linux-amd64-64/

Then when I tested my MR job, this time it gave back different error:

2013-10-04 16:32:01,539 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-10-04 16:32:01,869 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-10-04 16:32:01,877 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@b990b99
2013-10-04 16:32:02,017 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2013-10-04 16:32:02,024 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 256
2013-10-04 16:32:02,095 INFO org.apache.hadoop.mapred.MapTask: data buffer = 204010960/255013696
2013-10-04 16:32:02,095 INFO org.apache.hadoop.mapred.MapTask: record buffer = 671088/838860
2013-10-04 16:32:02,293 WARN Snappy native library is available
2013-10-04 16:32:02,293 INFO Snappy native library loaded
2013-10-04 16:32:23,909 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2013-10-04 16:32:24,342 WARN java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.initIDs()V
2013-10-04 16:32:24,342 INFO Got brand-new compressor
2013-10-04 16:32:24,403 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-10-04 16:32:24,418 INFO Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-10-04 16:32:24,418 INFO Got UserName yzhang for UID 1000 from the native implementation
2013-10-04 16:32:24,419 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I
        at org.apache.hadoop.mapred.IFile$Writer.close(
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(
        at org.apache.hadoop.mapred.MapTask.runOldMapper(
        at org.apache.hadoop.mapred.Child$
        at org.apache.hadoop.mapred.Child.main(

It looks like the hadoop knew that the native snappy is there, as shown  in the following log:

2013-10-04 16:32:02,293 WARN Snappy native library is available
2013-10-04 16:32:02,293 INFO Snappy native library loaded

But why it gave the above error?