Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
3 replies Latest Post - ‏2013-02-15T20:34:56Z by SystemAdmin
SystemAdmin
SystemAdmin
603 Posts
ACCEPTED ANSWER

Pinned topic console is not loading

‏2013-02-15T17:37:16Z |
Hi,
I am new to the BigInsights. Recently I installed the enterprise edition (v2) on a cluster of two machines for testing and learning purpose. By following the document provided at the online information center for BigInsights (v2), I tried to learn the machine data analysis by using the sample application provided within the package. After I installed the Accelerator for MDA, I have successfully deployed the data download application and the machine data analytics application through the BigInsights console, and also successfully run the jobs for downloading sample log data from provided location (watson) and completed the jobs for extracting log data and searching log records. Yesterday before I left my work I started run the job for identifying frequent sequences of errors (i.e.Time Window Transformation - Frequent Sequence Analysis), and this morning the console window is not working properly, after I refresh the window, it gave me error:

Exception thrown by application class 'com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.findWrapper():447'
JSPG0036E: Failed to find resource /BigInsights/console/NodeAdministration.jsp
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.findWrapper(AbstractJSPExtensionProcessor.java:447)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.getServletWrapper(AbstractJSPExtensionProcessor.java:368)
I don't know what went wrong. I searched through this forum and decided to run "stopall.sh" and then "start-all.sh", there were some errors and I still couldn't load the console. when I run stopall.sh, most of the components were stopped properly except for hadoop, here is part of the messages from the terminal:
....
.....
INFO Progress - 82%
INFO Progress - Stop hadoop
INFO @MyMasterNodeHostName - Tail /home/biadmin/local/var/ibm/biginsights/hadoop/logs/hadoop-biadmin-jobtracker-handmedown2.log
INFO @MyMasterNodeHostName - 2013-02-15 09:29:01,875 INFO com.ibm.biginsights.mapred.EnhancedJobTracker: SHUTDOWN_MSG:
INFO @MyMasterNodeHostName - /************************************************************
INFO @MyMasterNodeHostName - SHUTDOWN_MSG: Shutting down EnhancedJobTracker at MyMasterNodeHostName
INFO @MyMasterNodeHostName - ************************************************************/
INFO @MyMasterNodeHostName - ERROR jobtracker failed to stop
INFO Progress - 84%
INFO Progress - 91%
INFO Progress - Stop zookeeper
INFO @MyMasterNodeHostName - zookeeper stopped
INFO Deployer - zookeeper service stopped
INFO Progress - 100%
ERROR DeployManager -
ERROR DeployManager - OPERATION ERROR -- Stop console, derby, flume, hadoop, hbase, hive, jaqlserver, monitoring, oozie, orchestrator, zookeeper:
ERROR DeployManager -
INFO DeployManager - monitoring succeeded --
INFO DeployManager - console succeeded -- MyMasterNodeHostName=0
INFO DeployManager - jaqlserver succeeded -- MyMasterNodeHostName=0
INFO DeployManager - orchestrator succeeded -- MyMasterNodeHostName=0
INFO DeployManager - oozie succeeded -- MyMasterNodeHostName=0
INFO DeployManager - flume succeeded -- MyMasterNodeHostName=0
INFO DeployManager - hbase succeeded -- MyMasterNodeHostName=0
INFO DeployManager - hive succeeded -- MyMasterNodeHostName=0
INFO DeployManager - derby succeeded -- MyMasterNodeHostName=0
ERROR DeployManager - hadoop failed
com.ibm.xap.mgmt.DeployException: master node error -- MyMasterNodeHostName=1 (master fail! stop jobtracker failed because of: null)
at com.ibm.xap.mgmt.hdm.HadoopMgmtFacade.calculateResult(HadoopMgmtFacade.java:2170)
at com.ibm.xap.mgmt.hdm.HadoopMgmtFacade.daemon(HadoopMgmtFacade.java:609)
at com.ibm.xap.mgmt.hdm.HadoopMgmtFacade.daemon(HadoopMgmtFacade.java:589)
at com.ibm.xap.mgmt.hdm.HadoopDeployer.stopService(HadoopDeployer.java:551)
at com.ibm.xap.mgmt.DeployManager$StopThread.work(DeployManager.java:2531)
at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2332)
INFO DeployManager - zookeeper succeeded -- MyMasterNodeHostName=0
ERROR DeployManager - Stop; SUCCEEDED components: monitoring, console, jaqlserver, orchestrator, oozie, flume, hbase, hive, derby, zookeeper; FAILED components: hadoop; Consumes : 69992ms
Error exit.

I had no idea how to fix it, then I just run startall.sh, here are part of the messages I got:

...
...
INFO Progress - 82%
INFO Progress - Start console
INFO Deployer - starting pigserver, logging to /home/biadmin/local/var/ibm/biginsights/sheets/logs/bigsheets-biadmin-pigserver.out
INFO Deployer - /home/biadmin/local/opt/ibm/biginsights/console/wlp
INFO Deployer - before start-sheets
INFO Deployer - after start-sheets
INFO Deployer - pigserver running as process 11736. Stop it first.
INFO Deployer - rm: cannot remove `/home/biadmin/local/opt/ibm/biginsights/console/wlp/usr/servers/waslp-server/logs/console.log': No such file or directory
INFO Deployer - before start
INFO Deployer - Server waslp-server started with process ID 11890.
INFO Deployer - AUDIT CWWKF0011I: The server waslp-server is ready to run a smarter planet.
INFO Deployer - BigInsights Management Console started, pid 11890
INFO Progress - 91%
INFO Progress - Start monitoring
INFO Deployer - Start monitoring...
INFO Deployer - MyMasterNodeHostName: starting agent, logging to /home/biadmin/local/var/ibm/biginsights/monitoring/log/chukwa-biadmin-agent-handmedown2.out
MyDataNodeHostName: starting agent, logging to /home/biadmin/local/var/ibm/biginsights/monitoring/log/chukwa-biadmin-agent-handmedown3.beowulf.uwo.ca.out

INFO Progress - 100%
ERROR DeployManager -
ERROR DeployManager - OPERATION ERROR -- Start console, derby, flume, hadoop, hbase, hive, jaqlserver, monitoring, oozie, orchestrator, zookeeper:
ERROR DeployManager -
INFO DeployManager - zookeeper succeeded -- MyMasterNodeHostName=0 (9760-started)
INFO DeployManager - hadoop succeeded -- MyMasterNodeHostName=0; handmedown3.beowulf.uwo.ca=0
INFO DeployManager - derby succeeded -- MyMasterNodeHostName=0
ERROR DeployManager - hive failed
com.ibm.xap.mgmt.DeployException: Failed during parallel run tasks -- MyMasterNodeHostName=-1 (com.ibm.xap.mgmt.util.TimeoutException: Timeout when executing process, timeout = 300000
at com.ibm.xap.mgmt.util.Code.exec(Code.java:707)
at com.ibm.xap.mgmt.util.ExecUtil.exec(ExecUtil.java:84)
at com.ibm.xap.mgmt.util.ExecUtil.exec(ExecUtil.java:40)
at com.ibm.xap.mgmt.hive.HiveDeployer$5.doTask(HiveDeployer.java:1015)
at com.ibm.xap.mgmt.util.Task.run(Task.java:77)
at com.ibm.xap.mgmt.util.TaskRunner$1.run(TaskRunner.java:52)
)
at com.ibm.xap.mgmt.Deployer.parallelRunTasks(Deployer.java:956)
at com.ibm.xap.mgmt.hive.HiveDeployer.startService(HiveDeployer.java:1038)
at com.ibm.xap.mgmt.hive.HiveDeployer.startService(HiveDeployer.java:997)
at com.ibm.xap.mgmt.DeployManager$StartThread.work(DeployManager.java:2520)
at com.ibm.xap.mgmt.DeployManager$WorkerThread.run(DeployManager.java:2332)
INFO DeployManager - hbase succeeded -- MyDataNodeHostName=0
INFO DeployManager - flume succeeded -- MyMasterNodeHostName=0
INFO DeployManager - oozie succeeded -- MyMasterNodeHostName=0 (nullStarting Oozie WLP...
Server oozie started with process ID 11313.
.cat: /home/biadmin/local/opt/ibm/biginsights/oozie/wlp/usr/servers/oozie/logs/messages.log: No such file or directory
Oozie WLP started, pid 11313 .
)
INFO DeployManager - orchestrator succeeded -- MyMasterNodeHostName=0 (11507)
INFO DeployManager - jaqlserver succeeded -- MyMasterNodeHostName=0
INFO DeployManager - console succeeded -- MyMasterNodeHostName=0
INFO DeployManager - monitoring succeeded --
ERROR DeployManager - Start; SUCCEEDED components: zookeeper, hadoop, derby, hbase, flume, oozie, orchestrator, jaqlserver, console, monitoring; FAILED components: hive; Consumes : 396272ms
Error exit.

After all these, I still couldn't load the console. Is anybody could help me out here? I noticed a javacore file was generated after I left my work yesterday, I looked into the contents, it seems a native memory issue,i.e. "OutOfMemoryError", "Failed to create a thread".. I still have no idea how to fix it. I really hope that someone could give me some tips or suggestions, and I will appreciate it very much.

I attached the console.log file here and the javacore file to my next post and hope they can provide more information about the problem. Thanks!!

Jenny
Updated on 2013-02-15T20:34:56Z at 2013-02-15T20:34:56Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    603 Posts
    ACCEPTED ANSWER

    Re: console is not loading

    ‏2013-02-15T17:39:19Z  in response to SystemAdmin
    Here is the javacore file.
  • SystemAdmin
    SystemAdmin
    603 Posts
    ACCEPTED ANSWER

    Re: console is not loading

    ‏2013-02-15T19:22:17Z  in response to SystemAdmin
    Hi Jenny,
    This issue you are having may be related to resource limitations that apply to the user running the job.
    Confirm the BI administrator account is being used (or find out what
    account is used) and check resource limits. Check the settings in /etc/security/limits.conf.
    we recommend setting user file limit to 32000 for both. The following shows how to change it
    /etc/security/limits.conf:

    Link:

    http://pic.dhe.ibm.com/infocenter/bigins/v2r0/topic/com.ibm.swg.im.infosphere.biginsights.trb.doc/doc/troubleshooting_problems_admin.html

    Pasting here:

    Explanation: The current process exceeds the limit of file handles that can be open concurrently. The default file descriptor limit of 1024 for the InfoSphere BigInsights administrator is often insufficient.

    Workaround: As a system administrator, increase the default file descriptor of the InfoSphere BigInsights administrator to increase the user file limit:
    Check the current limit for the InfoSphere BigInsights Administrator user by entering the following command:
    $ulimit -n 1024
    Open the /etc/security/limits.conf file, and where biadmin is your InfoSphere BigInsights Administrator user, add the following lines:
    biadmin soft nofile 32000
    biadmin hard nofile 32000
    Restart the InfoSphere BigInsights Console to implement the file descriptor increase by entering the following command:
    $su - biadmin
    $/opt/ibm/biginsights/bin/stop.sh console
    $/opt/ibm/biginsights/bin/start.sh console

    See if this helps.

    Thanks,

    Zach
    • SystemAdmin
      SystemAdmin
      603 Posts
      ACCEPTED ANSWER

      Re: console is not loading

      ‏2013-02-15T20:34:56Z  in response to SystemAdmin
      Hi Zach,

      Thank you very much for the tips. Now it worked!!

      By the way, to load the web console, I only need to type localhost:8080/ it would then direct me to the console page at:
      http://localhost:8080/data/html/index.html#redirect-app?show=manage

      Another reason that I wasn't able to load the console previously was because I typed in "localhost:8080/BigInsights/.." which couldn't be resolved or directed to the actual console page ..

      Jenny