I've an application made to run both as a normal Java application and inside BigInsights, in the latter case saving the output in HDFS and getting the input from it.
This application, a web scraper, store in RAM the list of pending pages to be analyzed. This list can grow indefinitely so I thought to use a HBase table instead of a HashMap to store it.
When I run the application it checks for a HBase instance with
and seeing if this throws exceptions.
But the application don't even start with the error:
Launcher exception: org.apache.hadoop.hbase.MasterNotRunningException
it seems it can't find the class, which seems to be inside $BIGINSIGHTS_HOME/hbase/hbase-0.90.5.jar, so I changed biginsights-env.sh to add this jar and restarted the server, but the error is still there.
I tried to use the HBase example application to see how HBase is managed there (I've never used HBase before), but the src folder is empty.
Another problem is that to see the logs generated from the application I have to go to /var/ibm/biginsights/hadoop/logs and find the specific job folder and the right attempt folder, is it possible to see the log of an application directly from the web console ?
Thanks in advance
NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
This topic has been locked.
9 replies Latest Post - 2012-10-30T18:01:16Z by SystemAdmin
Pinned topic Use HBase under BIgInsights
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2012-10-30T18:01:16Z at 2012-10-30T18:01:16Z by SystemAdmin
Re: Use HBase under BIgInsights2012-10-25T18:27:45Z in response to SystemAdminCan you try the following and let me know
Check the status of hbase by running /opt/ibm/biginsights/bin/status.sh hbase -- This will give you the status of the hbase Master and Region Servers. If any of the servers are down, run the following command /opt/ibm/biginsights/bin/stop.sh hbase and then /opt/ibm/biginsights/bin/start.sh hbase
Re: Use HBase under BIgInsights2012-10-26T07:56:42Z in response to SystemAdminThe output is:
INFO Progress - Status hbase
INFO @localhost.localdomain - hbase-master(active) started
INFO @localhost.localdomain - hbase-regionserver started
INFO Progress - 100%
INFO DeployManager - Status; SUCCEEDED components: hbase]; FAILED components: [
However, I restarted with stop-al.sh and start-all.sh and even restarted the whole server (it's just a VM used for tests), but it still happens.
Re: Use HBase under BIgInsights2012-10-26T18:27:40Z in response to SystemAdminThanks Jacopo. If you have your application deployed on the Biginsights console, can you try the following.
1) Open your application on the web console
2) Click on the 'Configure' icon the top right corner. A configure dialog box will open.
3) Check if you have any path listed under RuntimeDependencies section. If nothing is listed, add the path to the hbase jar (hdfs path) there and try to run the app.
Re: Use HBase under BIgInsights2012-10-29T17:38:54Z in response to SystemAdminYou need not set the RuntimeDependencies each time you run the app. It is enough to set it once after you deploy the app.
Setting RuntimeDependencies can be done by either adding the path using the console UI or the other way is, through the application.xml that is part of the application bundle (zip file)
Inside the application.xml, you can set the RuntimeDependencies by including the path (comma separated) inside the <runtimeDependencies> tag as follows
Hope this helps.
Re: Use HBase under BIgInsights2012-10-29T17:57:42Z in response to SystemAdminThe following is an sample application.xml
<application-template xmlns="http://biginsights.ibm.com/application" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<description>The HBase app exports the table to csv format</description>
<property description="Enter the table name." isInputPath="false" isOutputPath="false" isRequired="true" label="Table Name" name="tableName" paramtype="STRING" uitype="textfield"/>
<property description="Enter the row key." isInputPath="false" isOutputPath="false" isRequired="true" label="Row Key" name="rowKey" paramtype="STRING" uitype="textfield"/>
<property description="Enter the number of records to fetch." isInputPath="false" isOutputPath="false" isRequired="true" label="Number of records to fetch" name="numRecords" paramtype="STRING" uitype="textfield"/>
<property defaultValue="csv" description="Output format of the data - csv, json" isInputPath="false" isOutputPath="false" isRequired="true" label="Output format" name="outputFormat" paramtype="LIST" uitype="textfield" validValues="csv, json"/>
<property description="Output directory where the results will be saved." isDataset="true" isInputPath="false" isOutputPath="true" isRequired="true" label="Output Directory" name="outputDir" paramtype="DIRECTORYPATH" uitype="textfield"/>
<asset id="HBase" type="WORKFLOW"/>
Note: The tags are case sensitive.