Topic
5 replies Latest Post - ‏2013-05-21T10:30:04Z by YangWeiWei
jmblaise
jmblaise
6 Posts
ACCEPTED ANSWER

Pinned topic Biginsights Basic Edition 2.0 failed question

‏2013-01-07T07:10:50Z |
Hello,

I am trying to install Biginsights Basic Edition 2.0 on a RHEL 5.5 64 bit VM, and the validation fails:

WARN Deployer - Failed to create private credstore file hive_keystore_pwd.prop in HDFS
...
ERROR DeployManager -
ERROR DeployManager - OPERATION ERROR -- Validate hadoop, hbase:
ERROR DeployManager -
ERROR DeployManager - hadoop failed
java.io.IOException: exit code: 2 -- "/opt/ibm/biginsights/hdm/bin/hdm" "checkdeploy"

Attached complete log.

I am selecting "cluster install" forcing the DataNode to be on localhost as the other nodes. The installer does not complain, but I wonder what is the problem. What is the minimum number of VMs needed for a "cluster install", can it be 1 ?

Thanks for your help,

JMB
Updated on 2013-01-08T20:11:49Z at 2013-01-08T20:11:49Z by jmblaise
  • jmblaise
    jmblaise
    6 Posts
    ACCEPTED ANSWER

    Re: Biginsights Basic Edition 2.0 failed question

    ‏2013-01-07T22:07:34Z  in response to jmblaise
    Hello,

    I removed /opm/ibm/biginsights and /var/ibm/biginsights directories, as I did not find an uninstaller shell.

    I reinstalled, picking up "single node" installation. The DataNode is said to be on "except nameNodes" machine ... quite confusing when using only localhost (I should have left the default when picking up Cluster installation).

    But the result is the same (except for ssh keys that are already there), failure to validate Hadoop.

    What is wrong ?

    Thanks,

    JMB
    • jmblaise
      jmblaise
      6 Posts
      ACCEPTED ANSWER

      Re: Biginsights Basic Edition 2.0 failed question

      ‏2013-01-07T22:18:46Z  in response to jmblaise
      Hello again,

      Following indication from the installation log,

      ERROR DeployManager -
      ERROR DeployManager - OPERATION ERROR -- Validate hadoop, hbase:
      ERROR DeployManager -
      ERROR DeployManager - hadoop failed
      java.io.IOException: exit code: 2 -- "/opt/ibm/biginsights/hdm/bin/hdm" "checkdeploy"

      I figured out that "/opt/ibm/biginsights/hdm/bin/hdm" "checkdeploy" refers to /opt/ibm/biginsights/hdm/bin/hdm-checkdeploy.sh script (it should be corrected in the installer).

      Now I have the same symptom "java.lang.NegativeArraySizeException".

      biadmin@bigdata bin$ ./hdm-checkdeploy.sh
      INFO Progress - Checking Hadoop cluster started
      INFO HadoopMgmtCmdline - Running daemon start
      INFO @localhost - namenode already running, pid 10026
      INFO @localhost - secondarynamenode already running, pid 10299
      INFO @localhost - datanode already running, pid 10471
      INFO @localhost - jobtracker already running, pid 10807
      INFO @localhost - tasktracker already running, pid 10981
      INFO Progress - Waiting for exit of safe mode
      INFO HadoopMgmtCmdline - Running safemode wait
      INFO Progress - Running terasort example
      >> /opt/ibm/biginsights/IHC/bin/hadoop dfs -rmr /hdm-tera-input /hdm-tera-output /hdm-tera-report
      Deleted hdfs://localhost:9000/hdm-tera-input
      rmr: cannot remove /hdm-tera-output: No such file or directory.
      rmr: cannot remove /hdm-tera-report: No such file or directory.
      >> /opt/ibm/biginsights/IHC/bin/hadoop jar /opt/ibm/biginsights/IHC/hadoop-examples-1.0.3.jar teragen -Dmapred.map.tasks=1 -Ddfs.block.size=536870912 -Dmapred.map.child.java.opts=-Xmx1024m -Dmapred.reduce.child.java.opts=-Xmx2048m 10000 /hdm-tera-input
      Generating 10000 using 1 maps with step of 10000
      13/01/07 23:17:12 INFO mapred.JobClient: Running job: job_201301072250_0003
      13/01/07 23:17:13 INFO mapred.JobClient: map 0% reduce 0%
      13/01/07 23:17:19 INFO mapred.JobClient: map 100% reduce 0%
      13/01/07 23:17:21 INFO mapred.JobClient: Job complete: job_201301072250_0003
      13/01/07 23:17:21 INFO mapred.JobClient: Counters: 19
      13/01/07 23:17:21 INFO mapred.JobClient: Job Counters
      13/01/07 23:17:21 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=7420
      13/01/07 23:17:21 INFO mapred.JobClient: Launched map tasks=1
      13/01/07 23:17:21 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
      13/01/07 23:17:21 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
      13/01/07 23:17:21 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
      13/01/07 23:17:21 INFO mapred.JobClient: FileSystemCounters
      13/01/07 23:17:21 INFO mapred.JobClient: FILE_BYTES_WRITTEN=22298
      13/01/07 23:17:21 INFO mapred.JobClient: HDFS_BYTES_READ=81
      13/01/07 23:17:21 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1000000
      13/01/07 23:17:21 INFO mapred.JobClient: File Output Format Counters
      13/01/07 23:17:21 INFO mapred.JobClient: Bytes Written=1000000
      13/01/07 23:17:21 INFO mapred.JobClient: Map-Reduce Framework
      13/01/07 23:17:21 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1267515392
      13/01/07 23:17:21 INFO mapred.JobClient: Map input bytes=10000
      13/01/07 23:17:21 INFO mapred.JobClient: Physical memory (bytes) snapshot=47591424
      13/01/07 23:17:21 INFO mapred.JobClient: Map output records=10000
      13/01/07 23:17:21 INFO mapred.JobClient: CPU time spent (ms)=290
      13/01/07 23:17:21 INFO mapred.JobClient: Map input records=10000
      13/01/07 23:17:21 INFO mapred.JobClient: Total committed heap usage (bytes)=5917696
      13/01/07 23:17:21 INFO mapred.JobClient: Spilled Records=0
      13/01/07 23:17:21 INFO mapred.JobClient: SPLIT_RAW_BYTES=81
      13/01/07 23:17:21 INFO mapred.JobClient: File Input Format Counters
      13/01/07 23:17:21 INFO mapred.JobClient: Bytes Read=0
      >> /opt/ibm/biginsights/IHC/bin/hadoop jar /opt/ibm/biginsights/IHC/hadoop-examples-1.0.3.jar terasort -Dmapred.reduce.tasks=0 -Ddfs.block.size=536870912 -Dmapred.map.child.java.opts=-Xmx1024m -Dmapred.reduce.child.java.opts=-Xmx2048m -Dio.sort.record.percent=0.17 /hdm-tera-input /hdm-tera-output
      13/01/07 23:17:23 INFO terasort.TeraSort: starting
      13/01/07 23:17:23 INFO mapred.FileInputFormat: Total input paths to process : 1
      13/01/07 23:17:24 INFO util.NativeCodeLoader: Loaded the native-hadoop library
      13/01/07 23:17:24 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
      13/01/07 23:17:24 INFO compress.CodecPool: Got brand-new compressor
      Making 0 from 10000 records
      Step size is Infinity
      java.lang.NegativeArraySizeException
      at org.apache.hadoop.examples.terasort.TeraInputFormat$TextSampler.createPartitions(TeraInputFormat.java:92)
      at org.apache.hadoop.examples.terasort.TeraInputFormat.writePartitionFile(TeraInputFormat.java:141)
      at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:243)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      >> /opt/ibm/biginsights/IHC/bin/hadoop jar /opt/ibm/biginsights/IHC/hadoop-examples-1.0.3.jar teravalidate -Dmapred.map.child.java.opts=-Xmx1024m -Dmapred.reduce.child.java.opts=-Xmx2048m /hdm-tera-output /hdm-tera-report
      13/01/07 23:17:27 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:9000/user/biadmin/.staging/job_201301072250_0004
      13/01/07 23:17:27 ERROR security.UserGroupInformation: PriviledgedActionException as:biadmin cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/hdm-tera-output
      org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/hdm-tera-output
      at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
      at org.apache.hadoop.examples.terasort.TeraInputFormat.getSplits(TeraInputFormat.java:209)
      at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:989)
      at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:981)
      at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
      at java.security.AccessController.doPrivileged(AccessController.java:310)
      at javax.security.auth.Subject.doAs(Subject.java:573)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
      at org.apache.hadoop.examples.terasort.TeraValidate.run(TeraValidate.java:145)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.examples.terasort.TeraValidate.main(TeraValidate.java:153)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
      at java.lang.reflect.Method.invoke(Method.java:611)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      >> /opt/ibm/biginsights/IHC/bin/hadoop dfs -ls /hdm-tera-report
      ls: Cannot access /hdm-tera-report: No such file or directory.
      INFO =============== Summary of Hadoop Installation ===============
      INFO TeraSort ..................................Failed

      Can you please advise how to proceed ?

      Thanks, JMB
      • YangWeiWei
        YangWeiWei
        57 Posts
        ACCEPTED ANSWER

        Re: Biginsights Basic Edition 2.0 failed question

        ‏2013-01-08T00:43:24Z  in response to jmblaise
        You may notice that entire install process succeed except for hadoop healthcheck, actually this is a bug in biginsights hadoop healthcheck script while it's single node install and the node has 1 core only, the formula the script used set number of reduce tasks for terasort to be 0 incorrectly. Fortunatly, this does not impact the use of the product.

        Can you ignore this error and go on using the product ? The bug will be fixed soon.
      • YangWeiWei
        YangWeiWei
        57 Posts
        ACCEPTED ANSWER

        Re: Biginsights Basic Edition 2.0 failed question

        ‏2013-05-21T10:30:04Z  in response to jmblaise

        opt/ibm/biginsights/IHC/bin/hadoop jar /opt/ibm/biginsights/IHC/hadoop-examples-1.0.3.jar terasort -Dmapred.reduce.tasks=0 -Ddfs.block.size=536870912 -Dmapred.map.child.java.opts=-Xmx1024m -Dmapred.reduce.child.java.opts=-Xmx2048m -Dio.sort.record.percent=0.17 /hdm-tera-input /hdm-tera-output

        Looks like the number of reduce task is set to zero wrongly, the formula used to calculate the reduce task number is based on 

        Average number of cores * number of tasktrackers * 0.5 * 0.9

        if you run with single server with single core, that equals to 1 * 1 * 0.5 * 0.9  = 0.45 = 0, so I say it is a bug. Anyway, you can ignore this error and keep using hadoop, it should be functional. If you really want it work, you can edit $BIGINSIGHTS_HOME/hdm/bin/hdm-terasort.sh, set 

        numReduces=1

        before any reference it has