Topic
  • 12 replies
  • Latest Post - ‏2013-12-06T21:21:25Z by huiyang@sfsu
huiyang@sfsu
huiyang@sfsu
8 Posts

Pinned topic Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

‏2013-12-03T01:36:12Z |

I created three instances (one master, two data nodes) using the "IBM InfoSphere BigInsights Basic 1.4 64b" image on the Academic Skills Cloud at the same data center. All three instances are shown "active", but I couldn't launch the bigInsights web console. Nor could I use the "start-all.sh" to start the MapReduce framework after having ssh-ed to the master node. I then checked the "logs" of the master node instance. The logs indicated that bigInsights was not installed at all:

Force BigInsights installation
Create install directory
Extract install media
Create silent install response file
NOTICE: install.xml.erb is already in the correct location (/tmp)
/etc/rc3.d/S50cloud-startup3.sh: line 405: ruby: command not found
Install BigInsights
/tmp/biginsights-basic-linux64_b20120604_2018/silent-install/silent-install.sh: line 33: /tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60/bin/java: No such file or directory
sed: can't read /tmp/biginsights-basic-linux64_b20120604_2018/silent-install/timestampfile: No such file or directory
--------------------------------------------------------------
Beginning the silent installation of BigInsights. For more information, see the log /tmp/biginsights-basic-linux64_b20120604_2018/silent-install/silent-install_.log.
--------------------------------------------------------------
[INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh /tmp/install.xml
[INFO] Distribution Vendor :
[INFO] Extract Java for idcuser...
[INFO] Progress - Initializing install properties
[INFO] Progress - 0%
[INFO] HadoopConf - Hadoop Configuration class is not on classpath
[Fatal Error] :-1:-1: Premature end of file.
Failed to parse install.xml: /tmp/install.xml
org.xml.sax.SAXParseException: Premature end of file.
    at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
    at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
    at com.ibm.biginsights.install.config.util.XMLUtility.getDocument(Unknown Source)
    at com.ibm.biginsights.install.config.InstallReader.readBasicInstallXML(Unknown Source)
    at com.ibm.biginsights.install.config.InstallReader.readFinalFullInstallXML(Unknown Source)
    at com.ibm.xap.mgmt.hdm.MgmtConfInitializer.parseInstallXml(MgmtConfInitializer.java:112)
at com.ibm.xap.mgmt.hdm.MgmtConfInitializer.main(MgmtConfInitializer.java:88)
[FATAL] Failed to generate hdm.properties
[FATAL] Failed to silent install BigInsights.

Please help!

Updated on 2013-12-03T07:17:43Z at 2013-12-03T07:17:43Z by huiyang@sfsu
  • DiLi
    DiLi
    69 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-03T13:14:08Z  

    Does /tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60 exist? If it doesn't, you need to run ./start.sh to have the installer prepare the ibm jvm for you, then you can go ahead and run the silent install.

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-04T01:22:56Z  
    • DiLi
    • ‏2013-12-03T13:14:08Z

    Does /tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60 exist? If it doesn't, you need to run ./start.sh to have the installer prepare the ibm jvm for you, then you can go ahead and run the silent install.

    Thank you for your reply.  I did run ./start.sh under the /tmp/biginsights-basic-linux64_b20120604_2018/   directory after I ssh-ed to the master node.  Below is the message.  The link provided below however is not working. 

     [idcuser@vhost1147 biginsights-basic-linux64_b20120604_2018]$ sudo ./start.sh
    Extracting Java ....
    Java extraction complete, using JAVA_HOME=/tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60
    Verifying port 8300 availability
    port 8300 available
    Starting BigInsights Installer .....
    BigInsights Installer started, please use a browser to access:
        http://170.224.164.127:8300/Install

    After you are finished, run the following command to stop the installer web server:
        start.sh shutdown

     

    After a while, I manually checked the directory /mnt/biginsights/. It is empty. I assume this means that the installation is not done properly. However, the  /tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60 directory does exist. Then I went ahead running the silent install (./silent-install.sh) anyway, and got the following error message:

    [idcuser@vhost1147 silent-install]$ ./silent-install.sh
    --------------------------------------------------------------
    Beginning the silent installation of BigInsights. For more information, see the log /tmp/biginsights-basic-linux64_b20120604_2018/silent-install/silent-install_2013-12-03T20.33.03.665_EST.log.
    --------------------------------------------------------------
    [INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh
    [FATAL] Cannot read /tmp/biginsights-basic-linux64_b20120604_2018/install.xml

    [FATAL] Failed to silent install BigInsights.

     

    Updated on 2013-12-04T01:35:48Z at 2013-12-04T01:35:48Z by huiyang@sfsu
  • DiLi
    DiLi
    69 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-04T13:42:33Z  

    Thank you for your reply.  I did run ./start.sh under the /tmp/biginsights-basic-linux64_b20120604_2018/   directory after I ssh-ed to the master node.  Below is the message.  The link provided below however is not working. 

     [idcuser@vhost1147 biginsights-basic-linux64_b20120604_2018]$ sudo ./start.sh
    Extracting Java ....
    Java extraction complete, using JAVA_HOME=/tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60
    Verifying port 8300 availability
    port 8300 available
    Starting BigInsights Installer .....
    BigInsights Installer started, please use a browser to access:
        http://170.224.164.127:8300/Install

    After you are finished, run the following command to stop the installer web server:
        start.sh shutdown

     

    After a while, I manually checked the directory /mnt/biginsights/. It is empty. I assume this means that the installation is not done properly. However, the  /tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60 directory does exist. Then I went ahead running the silent install (./silent-install.sh) anyway, and got the following error message:

    [idcuser@vhost1147 silent-install]$ ./silent-install.sh
    --------------------------------------------------------------
    Beginning the silent installation of BigInsights. For more information, see the log /tmp/biginsights-basic-linux64_b20120604_2018/silent-install/silent-install_2013-12-03T20.33.03.665_EST.log.
    --------------------------------------------------------------
    [INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh
    [FATAL] Cannot read /tmp/biginsights-basic-linux64_b20120604_2018/install.xml

    [FATAL] Failed to silent install BigInsights.

     

    All you did by running ./start.sh is to have the installer prepare JVM and then launch the installer UI. No install should've been kicked off since you didn't run the install via UI.

    You need to provide the path of the install.xml file as a parameter for the silent install script.

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-04T21:12:36Z  
    • DiLi
    • ‏2013-12-04T13:42:33Z

    All you did by running ./start.sh is to have the installer prepare JVM and then launch the installer UI. No install should've been kicked off since you didn't run the install via UI.

    You need to provide the path of the install.xml file as a parameter for the silent install script.

    Thanks! The installation UI failed to launch after running the command ./start.sh so I had no way to monitor the installation process.  For the silent install, I added the path to the install.xml. It failed again, because this xml file is empty. Any advice?

    Here's a different question: shouldn't biginsights be automatically installed if I create my instance using the "BigInsights Basic 1.4 64b" image?  

  • DiLi
    DiLi
    69 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-04T21:27:32Z  

    Thanks! The installation UI failed to launch after running the command ./start.sh so I had no way to monitor the installation process.  For the silent install, I added the path to the install.xml. It failed again, because this xml file is empty. Any advice?

    Here's a different question: shouldn't biginsights be automatically installed if I create my instance using the "BigInsights Basic 1.4 64b" image?  

    I don't have insights on how " using the "IBM InfoSphere BigInsights Basic 1.4 64b" image on the Academic Skills" works. What I was trying to guide you thru is to manually install a BI cluster with a basic edition installer.

    Your original post indicated your install was running until it hit a JVM issue, so your system must've had an install.xml somewhere. In fact, this is what was run based on your original post: [INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh /tmp/install.xml
    so the install.xml was in the /tmp dir.

    For silent install, you need to generate an install.xml file yourself, there are examples for various console security type, in the silent-install dir. When you said the installation UI failed to launch, can you provide details? If it's because 170.224.164.127 is a private ip, you can always switch it to a public ip/host. the one you use to putty to your linux system, for instance.It's much easier for you to kick off a BI install vis the installer UI as it takes care of generating the install.xml and writing install log to a log file.

     



     

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-04T21:55:44Z  
    • DiLi
    • ‏2013-12-04T21:27:32Z

    I don't have insights on how " using the "IBM InfoSphere BigInsights Basic 1.4 64b" image on the Academic Skills" works. What I was trying to guide you thru is to manually install a BI cluster with a basic edition installer.

    Your original post indicated your install was running until it hit a JVM issue, so your system must've had an install.xml somewhere. In fact, this is what was run based on your original post: [INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh /tmp/install.xml
    so the install.xml was in the /tmp dir.

    For silent install, you need to generate an install.xml file yourself, there are examples for various console security type, in the silent-install dir. When you said the installation UI failed to launch, can you provide details? If it's because 170.224.164.127 is a private ip, you can always switch it to a public ip/host. the one you use to putty to your linux system, for instance.It's much easier for you to kick off a BI install vis the installer UI as it takes care of generating the install.xml and writing install log to a log file.

     



     

    Thank you so much for your prompt response.  I'm fine installing BI manually as long as it succeeds.  (I assigned this as part of a class project for my students so they can use the Hadoop MapReduce to solve several problems.)

    I ran ./silent-install.sh on the master node of a different BI cluster with /tmp/install.xml and saw some progress, but the installation still doens't seem successful. (Note that I haven't done any installation on the two data nodes.) Here's the info shown on my screen. (Sorry it's super long.)

    [idcuser@vhost0736 silent-install]$ ./silent-install.sh /tmp/install.xml
    --------------------------------------------------------------
    Beginning the silent installation of BigInsights. For more information, see the log /tmp/biginsights-basic-linux64_b20120604_2018/silent-install/silent-install_2013-12-04T16.33.41.617_EST.log.
    --------------------------------------------------------------
    [INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh /tmp/install.xml
    [INFO] Distribution Vendor : ibm
    [INFO] Progress - Initializing install properties
    [INFO] Progress - 0%
    [INFO] HadoopConf - Hadoop Configuration class is not on classpath
    [INFO] MgmtConfInitializer - Biginsights nodes [170.224.194.226]
    [INFO] MgmtConfInitializer - install mode : install
    [INFO] MgmtConfInitializer - distro vendor : ibm
    [INFO] MgmtConfInitializer - dfs.name.dir=/mnt/biginsights/hadoop/hdfs/name
    [INFO] MgmtConfInitializer - fs.checkpoint.dir=/mnt/biginsights/hadoop/hdfs/namesecondary
    [INFO] MgmtConfInitializer - default dfs.data.dir=/mnt/biginsights/hadoop/hdfs/data
    [INFO] MgmtConfInitializer - mapred.system.dir=/hadoop/mapred/system
    [INFO] MgmtConfInitializer - mapred.local.dir=/mnt/biginsights/hadoop/mapred/local
    [INFO] MgmtConfInitializer - hadoop.log.dir=/mnt/biginsights/var/ibm/biginsights/hadoop/logs
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.hosts=*
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.groups=*
    [INFO] MgmtConfInitializer - datanode is not set
    [INFO] MgmtConfInitializer - 170.224.194.226 is NameNode
    [INFO] MgmtConfInitializer - 170.224.194.226 is Secondary NameNode
    [INFO] MgmtConfInitializer - 170.224.194.226 is JobTracker
    [INFO] MgmtConfInitializer - 170.224.194.226 is DataNode
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.hosts=*
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.groups=*
    [INFO] MgmtConfInitializer - Detect user group: idcuser
    [INFO] MgmtConfInitializer - biginsights.home=/mnt/biginsights/opt/ibm/biginsights
    [INFO] MgmtConfInitializer - biginsights.var=/mnt/biginsights/var/ibm/biginsights
    [INFO] MgmtConfInitializer - mgmt.ssh.config=assumed
    [INFO] MgmtConfInitializer - mgmt.user=idcuser
    [INFO] MgmtConfInitializer - mgmt.group=idcuser
    [INFO] MgmtConfInitializer - biginsights.virtualnodes=null
    [INFO] HadoopConf - Hadoop conf saved to /tmp/biginsights-basic-linux64_b20120604_2018/installer/hdm/hadoop-conf-staging
    [INFO] Progress - Check cluster environment
    [INFO] Progress - 2%
    [INFO] HadoopConf - Hadoop Configuration class is not on classpath
    [INFO] @170.224.194.226 - Check port <61616, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9999, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8009, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50090, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50070, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <54198, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50030, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60030, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50075, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50010, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60010, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9000, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9001, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <10080, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1527, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1528, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8080, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <10000, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8280, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50020, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60020, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60001, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <2181, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1050, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8200, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <6882, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <61613, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9997, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9998, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1099, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <2001, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <4201, 170.224.194.226> succeed.
    [INFO] Progress - 3%
    [INFO] PriorChecker - Pre-check on BigInsights Service Ports on Node Check : succeed.
    [INFO] Check the current user
    [INFO] Using... BIGINSIGHTS_HOME: /mnt/biginsights/opt/ibm/biginsights
    [INFO] Using... BIGINSIGHTS_VAR: /mnt/biginsights/var/ibm/biginsights
    [INFO] Using... SSH CONFIG MODE: assumed
    [INFO] Using... Biginsights administrator: idcuser
    [INFO] Progress - BigInsights installation response file type: install
    [INFO] HadoopConf - Hadoop Configuration class is not on classpath
    [INFO] Progress - Installing HDM
    [INFO] Progress - 3%
    [INFO] Progress - Preparing JDK package
    [INFO] Progress - 4%
    [INFO] JDK at /mnt/biginsights/opt/ibm/biginsights/hdm/jdk
    [INFO] Progress - Preparing Hadoop package
    [INFO] Progress - 6%
    [INFO] Hadoop at /mnt/biginsights/opt/ibm/biginsights/hdm/IHC
    [INFO] Progress - Configuring password-less SSH
    [INFO] Progress - 8%
    [INFO] HadoopMgmtCmdline - Running configAccountAndSsh /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/../../artifacts
    [INFO] DupHostDefender - Add other known names to  ~/.ssh/known_hosts.
    [INFO] Progress - 14%
    [INFO] Install as HADOOP_USER idcuser
    [INFO] Progress - Checking directories permission
    [INFO] Progress - 17%
    [INFO] HadoopMgmtCmdline - Running configDirs
    [INFO] @170.224.194.226 -
    [INFO] Progress - 18%
    [INFO] HadoopMgmtCmdline - Running check32or64
    [INFO] Progress - Deploying IBM Hadoop Cluster
    [INFO] Progress - 18%
    [INFO] HadoopMgmtCmdline - Running deployForceAll
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/hdm.tar.gz
    [INFO] HadoopConf - Reset includes (dfs.hosts)
    [INFO] HadoopConf - Reset includes (mapred.hosts)
    [INFO] HadoopConf - Auto set mapred.fairscheduler.allocation.file=/mnt/biginsights/opt/ibm/biginsights/hadoop-conf/fair-scheduler.xml
    [INFO] HadoopConf - Hadoop conf saved to /mnt/biginsights/opt/ibm/biginsights/hdm/hadoop-conf-staging
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/ihc-conf.tar.gz
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/ihc-conf.tar.gz@170.224.194.226
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/ihc.tar.gz
    [INFO] @170.224.194.226 - Deploy ... hdm, force
    [INFO] @170.224.194.226 - Deploy ... ihc-conf@170.224.194.226, force
    [INFO] @170.224.194.226 - Deploy ... ihc, force
    [INFO] @170.224.194.226 - Deploy ... jdk, force
    [INFO] @170.224.194.226 - Packages all up-to-date, after deploy [hdm, ihc-conf, ihc, jdk]
    [INFO] Progress - 25%
    [INFO] Progress - Clean up possible leftover process
    [INFO] Progress - 25%
    [INFO] HadoopMgmtCmdline - Running cleanupForInstall
    [INFO] @170.224.194.226 - /mnt/biginsights/opt/ibm/biginsights/.hdm-stub/bin/managed-cleanupForInstall.sh
    Cleanup for /mnt/biginsights/opt/ibm/biginsights
    Failed to kill /mnt/biginsights/opt/ibm/biginsights/, left 1554
    killed /mnt/biginsights/opt/ibm/biginsights/
    [INFO] Progress - Upgrading number of file handlers
    [INFO] Progress - 26%
    [INFO] HadoopMgmtCmdline - Running syncnofile 16384
    [WARN] Deployer - Skip syncNumFileHandler(), because no root password configured for nodes
    [INFO] Progress - Synchronizing system time
    [INFO] Progress - 27%
    [INFO] HadoopMgmtCmdline - Running synctime
    [WARN] Deployer - Skip syncTime(), because no root password configured for nodes
    [INFO] Progress - Installing BigInsights applications
    [INFO] Progress - 35%
    [FATAL] hdm was already installed!
    java.lang.RuntimeException: hdm was already installed!
        at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:601)
        at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:570)
        at com.ibm.xap.mgmt.DeployCmdline.runCommand(DeployCmdline.java:137)
        at com.ibm.xap.mgmt.DeployCmdline.run(DeployCmdline.java:101)
        at com.ibm.xap.mgmt.DeployCmdline.main(DeployCmdline.java:63)
    [INFO] Progress - Verifying BigInsights installation
    [INFO] Progress - 0%
    [INFO] Progress - Health check zookeeper
    [INFO] Progress - 9%
    [INFO] Progress - Health check hadoop
    [INFO] Deployer - Running Hadoop word count example
    [INFO] Deployer - hadoop service is healthy
    [INFO] Progress - 18%
    [INFO] Progress - Health check derby
    [INFO] Progress - 27%
    [INFO] Progress - Health check jaql
    [INFO] Progress - 36%
    [INFO] Progress - Health check hive
    [INFO] Progress - 45%
    [INFO] Progress - Health check pig
    [INFO] Progress - 55%
    [INFO] Progress - Health check hbase
    [INFO] Progress - 64%
    [INFO] Progress - Health check flume
    [INFO] Progress - 73%
    [INFO] Progress - Health check oozie
    [INFO] Progress - 82%
    [INFO] Progress - Health check jaqlserver
    [INFO] Progress - 91%
    [INFO] Progress - Health check console
    [INFO] Progress - 100%
    [ERROR] DeployManager -
    [ERROR] DeployManager - OPERATION ERROR -- Health check [hive, jaql, jaqlserver, hadoop, oozie, console, flume, derby, hbase, zookeeper, pig]:
    [ERROR] DeployManager - -------------------------------------------------------
    [ERROR] DeployManager - zookeeper failed
    java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/zookeeper/conf/zoo.cfg (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.loadConfig(ZookeeperDeployer.java:874)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.getDeployedNodes(ZookeeperDeployer.java:520)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.getNodes(ZookeeperDeployer.java:853)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.checkService(ZookeeperDeployer.java:682)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.healthCheck(ZookeeperDeployer.java:238)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [INFO] DeployManager - hadoop succeeded --
    [ERROR] DeployManager - derby failed
    com.ibm.xap.mgmt.ConfigurationException: No node defined, key biderby.node
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.getNode(CommonCatalogDeployer.java:269)
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.getNodes(CommonCatalogDeployer.java:281)
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.doStart(CommonCatalogDeployer.java:343)
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.healthCheck(CommonCatalogDeployer.java:191)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - jaql failed
    com.ibm.xap.mgmt.ConfigurationException: Failed to load biginsights-jaql.properties
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.loadProps(JaqlDeployer.java:604)
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.getfsType(JaqlDeployer.java:644)
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.healthCheck(JaqlDeployer.java:245)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    Caused by: java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/jaql/biginsights-jaql.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.loadProps(JaqlDeployer.java:598)
        ... 4 more
    [ERROR] DeployManager - hive failed
    com.ibm.xap.mgmt.ConfigurationException: Failed to load biginsights-hive.properties
        at com.ibm.xap.mgmt.hive.HiveDeployer.loadProps(HiveDeployer.java:846)
        at com.ibm.xap.mgmt.hive.HiveDeployer.getHiveProp(HiveDeployer.java:826)
        at com.ibm.xap.mgmt.hive.HiveDeployer.startService(HiveDeployer.java:1083)
        at com.ibm.xap.mgmt.hive.HiveDeployer.healthCheck(HiveDeployer.java:523)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    Caused by: java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/hive/biginsights-hive.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at com.ibm.xap.mgmt.hive.HiveDeployer.loadProps(HiveDeployer.java:841)
        ... 5 more
    [ERROR] DeployManager - pig failed
    com.ibm.xap.mgmt.DeployException: meet error when trying to get installed pig nodes: Can not read file /mnt/biginsights/opt/ibm/biginsights/hdm/components/pig/biginsights-pig.properties
        at com.ibm.xap.mgmt.pig.PigDeployer.healthCheck(PigDeployer.java:376)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - hbase failed
    java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/hbase/conf/install-info.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at java.io.FileReader.<init>(FileReader.java:52)
        at com.ibm.xap.mgmt.hbase.HBaseDeployer.getAllMasters(HBaseDeployer.java:1696)
        at com.ibm.xap.mgmt.hbase.HBaseDeployer.checkServiceInParallel(HBaseDeployer.java:2125)
        at com.ibm.xap.mgmt.hbase.HBaseDeployer.healthCheck(HBaseDeployer.java:899)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - flume failed
    com.ibm.xap.mgmt.ConfigurationException: meet error during getDeployNodes:/mnt/biginsights/opt/ibm/biginsights/hdm/components/flume/conf/flume-conf.xml (No such file or directory)
        at com.ibm.xap.mgmt.flume.FlumeDeployer.getDeployedNodes(FlumeDeployer.java:2542)
        at com.ibm.xap.mgmt.flume.FlumeDeployer.healthCheck(FlumeDeployer.java:2166)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - oozie failed
    java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/oozie/conf/biginsights-oozie.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at java.io.FileReader.<init>(FileReader.java:52)
        at com.ibm.xap.mgmt.oozie.OozieDeployer.getPort(OozieDeployer.java:1024)
        at com.ibm.xap.mgmt.oozie.OozieDeployer.healthCheck(OozieDeployer.java:223)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - jaqlserver failed
    com.ibm.xap.mgmt.ConfigurationException: Failed to load biginsights-js.properties
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.loadProps(JaqlServerDeployer.java:639)
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.getNodes(JaqlServerDeployer.java:609)
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.healthCheck(JaqlServerDeployer.java:230)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    Caused by: java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/jaqlserver/biginsights-js.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.loadProps(JaqlServerDeployer.java:633)
        ... 4 more
    [INFO] DeployManager - console succeeded --
    [ERROR] DeployManager - Health check; SUCCEEDED components: [hadoop, console]; FAILED components: [zookeeper, derby, jaql, hive, pig, hbase, flume, oozie, jaqlserver]
    Error exit.
    [FATAL] Failed to install BigInsights component(s)

    [FATAL] Failed to silent install BigInsights.

     

  • DiLi
    DiLi
    69 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-04T22:22:43Z  

    Thank you so much for your prompt response.  I'm fine installing BI manually as long as it succeeds.  (I assigned this as part of a class project for my students so they can use the Hadoop MapReduce to solve several problems.)

    I ran ./silent-install.sh on the master node of a different BI cluster with /tmp/install.xml and saw some progress, but the installation still doens't seem successful. (Note that I haven't done any installation on the two data nodes.) Here's the info shown on my screen. (Sorry it's super long.)

    [idcuser@vhost0736 silent-install]$ ./silent-install.sh /tmp/install.xml
    --------------------------------------------------------------
    Beginning the silent installation of BigInsights. For more information, see the log /tmp/biginsights-basic-linux64_b20120604_2018/silent-install/silent-install_2013-12-04T16.33.41.617_EST.log.
    --------------------------------------------------------------
    [INFO] Running as idcuser, /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/install.sh /tmp/install.xml
    [INFO] Distribution Vendor : ibm
    [INFO] Progress - Initializing install properties
    [INFO] Progress - 0%
    [INFO] HadoopConf - Hadoop Configuration class is not on classpath
    [INFO] MgmtConfInitializer - Biginsights nodes [170.224.194.226]
    [INFO] MgmtConfInitializer - install mode : install
    [INFO] MgmtConfInitializer - distro vendor : ibm
    [INFO] MgmtConfInitializer - dfs.name.dir=/mnt/biginsights/hadoop/hdfs/name
    [INFO] MgmtConfInitializer - fs.checkpoint.dir=/mnt/biginsights/hadoop/hdfs/namesecondary
    [INFO] MgmtConfInitializer - default dfs.data.dir=/mnt/biginsights/hadoop/hdfs/data
    [INFO] MgmtConfInitializer - mapred.system.dir=/hadoop/mapred/system
    [INFO] MgmtConfInitializer - mapred.local.dir=/mnt/biginsights/hadoop/mapred/local
    [INFO] MgmtConfInitializer - hadoop.log.dir=/mnt/biginsights/var/ibm/biginsights/hadoop/logs
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.hosts=*
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.groups=*
    [INFO] MgmtConfInitializer - datanode is not set
    [INFO] MgmtConfInitializer - 170.224.194.226 is NameNode
    [INFO] MgmtConfInitializer - 170.224.194.226 is Secondary NameNode
    [INFO] MgmtConfInitializer - 170.224.194.226 is JobTracker
    [INFO] MgmtConfInitializer - 170.224.194.226 is DataNode
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.hosts=*
    [INFO] MgmtConfInitializer - hadoop.proxyuser.idcuser.groups=*
    [INFO] MgmtConfInitializer - Detect user group: idcuser
    [INFO] MgmtConfInitializer - biginsights.home=/mnt/biginsights/opt/ibm/biginsights
    [INFO] MgmtConfInitializer - biginsights.var=/mnt/biginsights/var/ibm/biginsights
    [INFO] MgmtConfInitializer - mgmt.ssh.config=assumed
    [INFO] MgmtConfInitializer - mgmt.user=idcuser
    [INFO] MgmtConfInitializer - mgmt.group=idcuser
    [INFO] MgmtConfInitializer - biginsights.virtualnodes=null
    [INFO] HadoopConf - Hadoop conf saved to /tmp/biginsights-basic-linux64_b20120604_2018/installer/hdm/hadoop-conf-staging
    [INFO] Progress - Check cluster environment
    [INFO] Progress - 2%
    [INFO] HadoopConf - Hadoop Configuration class is not on classpath
    [INFO] @170.224.194.226 - Check port <61616, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9999, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8009, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50090, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50070, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <54198, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50030, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60030, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50075, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50010, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60010, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9000, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9001, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <10080, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1527, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1528, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8080, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <10000, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8280, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <50020, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60020, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <60001, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <2181, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1050, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <8200, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <6882, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <61613, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9997, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <9998, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <1099, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <2001, 170.224.194.226> succeed.
    [INFO] @170.224.194.226 - Check port <4201, 170.224.194.226> succeed.
    [INFO] Progress - 3%
    [INFO] PriorChecker - Pre-check on BigInsights Service Ports on Node Check : succeed.
    [INFO] Check the current user
    [INFO] Using... BIGINSIGHTS_HOME: /mnt/biginsights/opt/ibm/biginsights
    [INFO] Using... BIGINSIGHTS_VAR: /mnt/biginsights/var/ibm/biginsights
    [INFO] Using... SSH CONFIG MODE: assumed
    [INFO] Using... Biginsights administrator: idcuser
    [INFO] Progress - BigInsights installation response file type: install
    [INFO] HadoopConf - Hadoop Configuration class is not on classpath
    [INFO] Progress - Installing HDM
    [INFO] Progress - 3%
    [INFO] Progress - Preparing JDK package
    [INFO] Progress - 4%
    [INFO] JDK at /mnt/biginsights/opt/ibm/biginsights/hdm/jdk
    [INFO] Progress - Preparing Hadoop package
    [INFO] Progress - 6%
    [INFO] Hadoop at /mnt/biginsights/opt/ibm/biginsights/hdm/IHC
    [INFO] Progress - Configuring password-less SSH
    [INFO] Progress - 8%
    [INFO] HadoopMgmtCmdline - Running configAccountAndSsh /tmp/biginsights-basic-linux64_b20120604_2018/installer/bin/../../artifacts
    [INFO] DupHostDefender - Add other known names to  ~/.ssh/known_hosts.
    [INFO] Progress - 14%
    [INFO] Install as HADOOP_USER idcuser
    [INFO] Progress - Checking directories permission
    [INFO] Progress - 17%
    [INFO] HadoopMgmtCmdline - Running configDirs
    [INFO] @170.224.194.226 -
    [INFO] Progress - 18%
    [INFO] HadoopMgmtCmdline - Running check32or64
    [INFO] Progress - Deploying IBM Hadoop Cluster
    [INFO] Progress - 18%
    [INFO] HadoopMgmtCmdline - Running deployForceAll
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/hdm.tar.gz
    [INFO] HadoopConf - Reset includes (dfs.hosts)
    [INFO] HadoopConf - Reset includes (mapred.hosts)
    [INFO] HadoopConf - Auto set mapred.fairscheduler.allocation.file=/mnt/biginsights/opt/ibm/biginsights/hadoop-conf/fair-scheduler.xml
    [INFO] HadoopConf - Hadoop conf saved to /mnt/biginsights/opt/ibm/biginsights/hdm/hadoop-conf-staging
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/ihc-conf.tar.gz
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/ihc-conf.tar.gz@170.224.194.226
    [INFO] Cluster - Repack ... /mnt/biginsights/opt/ibm/biginsights/hdm/todeploy/ihc.tar.gz
    [INFO] @170.224.194.226 - Deploy ... hdm, force
    [INFO] @170.224.194.226 - Deploy ... ihc-conf@170.224.194.226, force
    [INFO] @170.224.194.226 - Deploy ... ihc, force
    [INFO] @170.224.194.226 - Deploy ... jdk, force
    [INFO] @170.224.194.226 - Packages all up-to-date, after deploy [hdm, ihc-conf, ihc, jdk]
    [INFO] Progress - 25%
    [INFO] Progress - Clean up possible leftover process
    [INFO] Progress - 25%
    [INFO] HadoopMgmtCmdline - Running cleanupForInstall
    [INFO] @170.224.194.226 - /mnt/biginsights/opt/ibm/biginsights/.hdm-stub/bin/managed-cleanupForInstall.sh
    Cleanup for /mnt/biginsights/opt/ibm/biginsights
    Failed to kill /mnt/biginsights/opt/ibm/biginsights/, left 1554
    killed /mnt/biginsights/opt/ibm/biginsights/
    [INFO] Progress - Upgrading number of file handlers
    [INFO] Progress - 26%
    [INFO] HadoopMgmtCmdline - Running syncnofile 16384
    [WARN] Deployer - Skip syncNumFileHandler(), because no root password configured for nodes
    [INFO] Progress - Synchronizing system time
    [INFO] Progress - 27%
    [INFO] HadoopMgmtCmdline - Running synctime
    [WARN] Deployer - Skip syncTime(), because no root password configured for nodes
    [INFO] Progress - Installing BigInsights applications
    [INFO] Progress - 35%
    [FATAL] hdm was already installed!
    java.lang.RuntimeException: hdm was already installed!
        at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:601)
        at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:570)
        at com.ibm.xap.mgmt.DeployCmdline.runCommand(DeployCmdline.java:137)
        at com.ibm.xap.mgmt.DeployCmdline.run(DeployCmdline.java:101)
        at com.ibm.xap.mgmt.DeployCmdline.main(DeployCmdline.java:63)
    [INFO] Progress - Verifying BigInsights installation
    [INFO] Progress - 0%
    [INFO] Progress - Health check zookeeper
    [INFO] Progress - 9%
    [INFO] Progress - Health check hadoop
    [INFO] Deployer - Running Hadoop word count example
    [INFO] Deployer - hadoop service is healthy
    [INFO] Progress - 18%
    [INFO] Progress - Health check derby
    [INFO] Progress - 27%
    [INFO] Progress - Health check jaql
    [INFO] Progress - 36%
    [INFO] Progress - Health check hive
    [INFO] Progress - 45%
    [INFO] Progress - Health check pig
    [INFO] Progress - 55%
    [INFO] Progress - Health check hbase
    [INFO] Progress - 64%
    [INFO] Progress - Health check flume
    [INFO] Progress - 73%
    [INFO] Progress - Health check oozie
    [INFO] Progress - 82%
    [INFO] Progress - Health check jaqlserver
    [INFO] Progress - 91%
    [INFO] Progress - Health check console
    [INFO] Progress - 100%
    [ERROR] DeployManager -
    [ERROR] DeployManager - OPERATION ERROR -- Health check [hive, jaql, jaqlserver, hadoop, oozie, console, flume, derby, hbase, zookeeper, pig]:
    [ERROR] DeployManager - -------------------------------------------------------
    [ERROR] DeployManager - zookeeper failed
    java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/zookeeper/conf/zoo.cfg (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.loadConfig(ZookeeperDeployer.java:874)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.getDeployedNodes(ZookeeperDeployer.java:520)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.getNodes(ZookeeperDeployer.java:853)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.checkService(ZookeeperDeployer.java:682)
        at com.ibm.xap.mgmt.zookeeper.ZookeeperDeployer.healthCheck(ZookeeperDeployer.java:238)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [INFO] DeployManager - hadoop succeeded --
    [ERROR] DeployManager - derby failed
    com.ibm.xap.mgmt.ConfigurationException: No node defined, key biderby.node
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.getNode(CommonCatalogDeployer.java:269)
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.getNodes(CommonCatalogDeployer.java:281)
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.doStart(CommonCatalogDeployer.java:343)
        at com.ibm.xap.mgmt.commoncatalog.CommonCatalogDeployer.healthCheck(CommonCatalogDeployer.java:191)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - jaql failed
    com.ibm.xap.mgmt.ConfigurationException: Failed to load biginsights-jaql.properties
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.loadProps(JaqlDeployer.java:604)
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.getfsType(JaqlDeployer.java:644)
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.healthCheck(JaqlDeployer.java:245)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    Caused by: java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/jaql/biginsights-jaql.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at com.ibm.xap.mgmt.jaql.JaqlDeployer.loadProps(JaqlDeployer.java:598)
        ... 4 more
    [ERROR] DeployManager - hive failed
    com.ibm.xap.mgmt.ConfigurationException: Failed to load biginsights-hive.properties
        at com.ibm.xap.mgmt.hive.HiveDeployer.loadProps(HiveDeployer.java:846)
        at com.ibm.xap.mgmt.hive.HiveDeployer.getHiveProp(HiveDeployer.java:826)
        at com.ibm.xap.mgmt.hive.HiveDeployer.startService(HiveDeployer.java:1083)
        at com.ibm.xap.mgmt.hive.HiveDeployer.healthCheck(HiveDeployer.java:523)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    Caused by: java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/hive/biginsights-hive.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at com.ibm.xap.mgmt.hive.HiveDeployer.loadProps(HiveDeployer.java:841)
        ... 5 more
    [ERROR] DeployManager - pig failed
    com.ibm.xap.mgmt.DeployException: meet error when trying to get installed pig nodes: Can not read file /mnt/biginsights/opt/ibm/biginsights/hdm/components/pig/biginsights-pig.properties
        at com.ibm.xap.mgmt.pig.PigDeployer.healthCheck(PigDeployer.java:376)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - hbase failed
    java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/hbase/conf/install-info.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at java.io.FileReader.<init>(FileReader.java:52)
        at com.ibm.xap.mgmt.hbase.HBaseDeployer.getAllMasters(HBaseDeployer.java:1696)
        at com.ibm.xap.mgmt.hbase.HBaseDeployer.checkServiceInParallel(HBaseDeployer.java:2125)
        at com.ibm.xap.mgmt.hbase.HBaseDeployer.healthCheck(HBaseDeployer.java:899)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - flume failed
    com.ibm.xap.mgmt.ConfigurationException: meet error during getDeployNodes:/mnt/biginsights/opt/ibm/biginsights/hdm/components/flume/conf/flume-conf.xml (No such file or directory)
        at com.ibm.xap.mgmt.flume.FlumeDeployer.getDeployedNodes(FlumeDeployer.java:2542)
        at com.ibm.xap.mgmt.flume.FlumeDeployer.healthCheck(FlumeDeployer.java:2166)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - oozie failed
    java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/oozie/conf/biginsights-oozie.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at java.io.FileReader.<init>(FileReader.java:52)
        at com.ibm.xap.mgmt.oozie.OozieDeployer.getPort(OozieDeployer.java:1024)
        at com.ibm.xap.mgmt.oozie.OozieDeployer.healthCheck(OozieDeployer.java:223)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    [ERROR] DeployManager - jaqlserver failed
    com.ibm.xap.mgmt.ConfigurationException: Failed to load biginsights-js.properties
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.loadProps(JaqlServerDeployer.java:639)
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.getNodes(JaqlServerDeployer.java:609)
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.healthCheck(JaqlServerDeployer.java:230)
        at com.ibm.xap.mgmt.DeployManager$HealthCheckThread.work(DeployManager.java:2348)
        at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2112)
    Caused by: java.io.FileNotFoundException: /mnt/biginsights/opt/ibm/biginsights/hdm/components/jaqlserver/biginsights-js.properties (No such file or directory)
        at java.io.FileInputStream.<init>(FileInputStream.java:137)
        at java.io.FileInputStream.<init>(FileInputStream.java:96)
        at com.ibm.xap.mgmt.jaqlserver.JaqlServerDeployer.loadProps(JaqlServerDeployer.java:633)
        ... 4 more
    [INFO] DeployManager - console succeeded --
    [ERROR] DeployManager - Health check; SUCCEEDED components: [hadoop, console]; FAILED components: [zookeeper, derby, jaql, hive, pig, hbase, flume, oozie, jaqlserver]
    Error exit.
    [FATAL] Failed to install BigInsights component(s)

    [FATAL] Failed to silent install BigInsights.

     

    the installation log for your other cluster showed the cluster had a bi instance installed with the same configurations specified in the /tmp/install.xml, here is the exception from the log:

    [FATAL] hdm was already installed!
    java.lang.RuntimeException: hdm was already installed!
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:601)
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:570)
    at com.ibm.xap.mgmt.DeployCmdline.runCommand(DeployCmdline.java:137)
    at com.ibm.xap.mgmt.DeployCmdline.run(DeployCmdline.java:101)
    at com.ibm.xap.mgmt.DeployCmdline.main(DeployCmdline.java:63)

    Your other cluster could have a working BI installation, or just the residual of previously failed installations that interferred with your latest install attempt.

    I strongly suggest you to try the installer UI, unless you can show me details on how the installer UI failed on you. The ./start.sh log you posted earlier indicated the installer UI was started successfully, you may also check port 8300 see if it's being used by the BI installer server. I've mentioned how you may access the installer UI, in my previous comment.

    Alternatively, if you opt to continue with the silent install, you can copy the /tmp/install.xml from your secondary cluster to your main cluster, and UPDATE it with the correct host name /ip , you should then be able to run silent install on the main cluster, it saves you from creating install.xml from scratch for your main cluster.

    Keep it in mind that before kicking off an install, you will probably need to remove files left by the previously failed BI  installations, please check the install.xml for dirs used by BI, it should includes $BIGINSIGHTS_HOME, $BIGINSIGHTS_VAR, hadoop directories at the minimum.

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-05T00:10:19Z  
    • DiLi
    • ‏2013-12-04T22:22:43Z

    the installation log for your other cluster showed the cluster had a bi instance installed with the same configurations specified in the /tmp/install.xml, here is the exception from the log:

    [FATAL] hdm was already installed!
    java.lang.RuntimeException: hdm was already installed!
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:601)
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:570)
    at com.ibm.xap.mgmt.DeployCmdline.runCommand(DeployCmdline.java:137)
    at com.ibm.xap.mgmt.DeployCmdline.run(DeployCmdline.java:101)
    at com.ibm.xap.mgmt.DeployCmdline.main(DeployCmdline.java:63)

    Your other cluster could have a working BI installation, or just the residual of previously failed installations that interferred with your latest install attempt.

    I strongly suggest you to try the installer UI, unless you can show me details on how the installer UI failed on you. The ./start.sh log you posted earlier indicated the installer UI was started successfully, you may also check port 8300 see if it's being used by the BI installer server. I've mentioned how you may access the installer UI, in my previous comment.

    Alternatively, if you opt to continue with the silent install, you can copy the /tmp/install.xml from your secondary cluster to your main cluster, and UPDATE it with the correct host name /ip , you should then be able to run silent install on the main cluster, it saves you from creating install.xml from scratch for your main cluster.

    Keep it in mind that before kicking off an install, you will probably need to remove files left by the previously failed BI  installations, please check the install.xml for dirs used by BI, it should includes $BIGINSIGHTS_HOME, $BIGINSIGHTS_VAR, hadoop directories at the minimum.

    Since I created the instance using the biginsights-1.4 image, this would result in the first and default installation attempt of BI. Anyway, here's what the screen displayed when I run /tmp/biginsights-basic-linux64_b20120604_2018/start.sh on the master node. 

    [idcuser@vhost1147 biginsights-basic-linux64_b20120604_2018]$ ./start.sh
    Extracting Java ....
    Java extraction complete, using JAVA_HOME=/tmp/biginsights-basic-linux64_b20120604_2018/_jvm/ibm-java-x86_64-60
    Verifying port 8300 availability
    port 8300 available
    Starting BigInsights Installer .....
    BigInsights Installer started, please use a browser to access:
        http://170.224.164.127:8300/Install

    After you are finished, run the following command to stop the installer web server:
        start.sh shutdown

     

    I clicked the above link http://170.224.164.127:8300/Install, but the link doesn't work. The browser says "The connection has timed out. The server at 170.224.164.127 is taking too long to respond.".  Yes, this IP address is public and I can access it through ssh. 

     

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-05T01:12:40Z  
    • DiLi
    • ‏2013-12-04T22:22:43Z

    the installation log for your other cluster showed the cluster had a bi instance installed with the same configurations specified in the /tmp/install.xml, here is the exception from the log:

    [FATAL] hdm was already installed!
    java.lang.RuntimeException: hdm was already installed!
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:601)
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:570)
    at com.ibm.xap.mgmt.DeployCmdline.runCommand(DeployCmdline.java:137)
    at com.ibm.xap.mgmt.DeployCmdline.run(DeployCmdline.java:101)
    at com.ibm.xap.mgmt.DeployCmdline.main(DeployCmdline.java:63)

    Your other cluster could have a working BI installation, or just the residual of previously failed installations that interferred with your latest install attempt.

    I strongly suggest you to try the installer UI, unless you can show me details on how the installer UI failed on you. The ./start.sh log you posted earlier indicated the installer UI was started successfully, you may also check port 8300 see if it's being used by the BI installer server. I've mentioned how you may access the installer UI, in my previous comment.

    Alternatively, if you opt to continue with the silent install, you can copy the /tmp/install.xml from your secondary cluster to your main cluster, and UPDATE it with the correct host name /ip , you should then be able to run silent install on the main cluster, it saves you from creating install.xml from scratch for your main cluster.

    Keep it in mind that before kicking off an install, you will probably need to remove files left by the previously failed BI  installations, please check the install.xml for dirs used by BI, it should includes $BIGINSIGHTS_HOME, $BIGINSIGHTS_VAR, hadoop directories at the minimum.

    Thought this might help you diagnose the problem. Attached is the install.xml that I used for the silent-install that gave me a partial success.  I notice that nowhere in this file the two data nodes are mentioned.  My first node IP is: 170.224.194.153 and the second IP  170.224.194.219.  Shall I modify this XML file to add these two data nodes?  If so, can you please give me an example?  Thanks.

     

    Attachments

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-05T02:27:08Z  
    • DiLi
    • ‏2013-12-04T22:22:43Z

    the installation log for your other cluster showed the cluster had a bi instance installed with the same configurations specified in the /tmp/install.xml, here is the exception from the log:

    [FATAL] hdm was already installed!
    java.lang.RuntimeException: hdm was already installed!
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:601)
    at com.ibm.xap.mgmt.DeployManager.install(DeployManager.java:570)
    at com.ibm.xap.mgmt.DeployCmdline.runCommand(DeployCmdline.java:137)
    at com.ibm.xap.mgmt.DeployCmdline.run(DeployCmdline.java:101)
    at com.ibm.xap.mgmt.DeployCmdline.main(DeployCmdline.java:63)

    Your other cluster could have a working BI installation, or just the residual of previously failed installations that interferred with your latest install attempt.

    I strongly suggest you to try the installer UI, unless you can show me details on how the installer UI failed on you. The ./start.sh log you posted earlier indicated the installer UI was started successfully, you may also check port 8300 see if it's being used by the BI installer server. I've mentioned how you may access the installer UI, in my previous comment.

    Alternatively, if you opt to continue with the silent install, you can copy the /tmp/install.xml from your secondary cluster to your main cluster, and UPDATE it with the correct host name /ip , you should then be able to run silent install on the main cluster, it saves you from creating install.xml from scratch for your main cluster.

    Keep it in mind that before kicking off an install, you will probably need to remove files left by the previously failed BI  installations, please check the install.xml for dirs used by BI, it should includes $BIGINSIGHTS_HOME, $BIGINSIGHTS_VAR, hadoop directories at the minimum.

    I was able to successful silent-install bi on the master node of another cluster by manually replacing the IP address in install.xml as advised by you. (Here's my web console: http://vhost1147.site1.compute.ihost.com:8080/BigInsights/console/NodeAdministration.jsp)

    To add data nodes to this cluster,   I used the "add node" function in the web console.  This seems working.  Am I missing anything? 

    Many thanks.

    Updated on 2013-12-05T07:09:55Z at 2013-12-05T07:09:55Z by huiyang@sfsu
  • DiLi
    DiLi
    69 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-05T13:44:28Z  

    I was able to successful silent-install bi on the master node of another cluster by manually replacing the IP address in install.xml as advised by you. (Here's my web console: http://vhost1147.site1.compute.ihost.com:8080/BigInsights/console/NodeAdministration.jsp)

    To add data nodes to this cluster,   I used the "add node" function in the web console.  This seems working.  Am I missing anything? 

    Many thanks.

    Did add node via console UI work or not? You can run "listnode.sh" as the biadmin user on the console node to see the nodes included in the bi cluster.

    you can also run addnode.sh to add node to your cluster, via cli. Here is the IC link
    http://pic.dhe.ibm.com/infocenter/bigins/v1r4/topic/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/r0057945.html?resultof=%22addnode.sh%22

  • huiyang@sfsu
    huiyang@sfsu
    8 Posts

    Re: Failed to install BigInsights using the "BigInsights Basic 1.4 64b" image

    ‏2013-12-06T21:21:25Z  
    • DiLi
    • ‏2013-12-05T13:44:28Z

    Did add node via console UI work or not? You can run "listnode.sh" as the biadmin user on the console node to see the nodes included in the bi cluster.

    you can also run addnode.sh to add node to your cluster, via cli. Here is the IC link
    http://pic.dhe.ibm.com/infocenter/bigins/v1r4/topic/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/r0057945.html?resultof=%22addnode.sh%22

    Yes, nodes have been successfully added to the hadoop cluster using the web console, although I much prefer running addnode.sh on the command line as it displays the installation status.  I am preparing a step-by-step instruction to share with my students on how to create a hadoop cluster on the Academic Skills Cloud. 

    Thank you so much for your time!

    Updated on 2013-12-06T22:53:35Z at 2013-12-06T22:53:35Z by huiyang@sfsu