Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
9 replies Latest Post - ‏2012-10-30T18:01:16Z by SystemAdmin
SystemAdmin
SystemAdmin
603 Posts
ACCEPTED ANSWER

Pinned topic Use HBase under BIgInsights

‏2012-10-24T11:22:06Z |
Hello,
I've an application made to run both as a normal Java application and inside BigInsights, in the latter case saving the output in HDFS and getting the input from it.
This application, a web scraper, store in RAM the list of pending pages to be analyzed. This list can grow indefinitely so I thought to use a HBase table instead of a HashMap to store it.
When I run the application it checks for a HBase instance with

HBaseAdmin.checkHBaseAvailable(HBaseConfiguration.create());

and seeing if this throws exceptions.

But the application don't even start with the error:

Launcher exception: org.apache.hadoop.hbase.MasterNotRunningException

it seems it can't find the class, which seems to be inside $BIGINSIGHTS_HOME/hbase/hbase-0.90.5.jar, so I changed biginsights-env.sh to add this jar and restarted the server, but the error is still there.

I tried to use the HBase example application to see how HBase is managed there (I've never used HBase before), but the src folder is empty.

Another problem is that to see the logs generated from the application I have to go to /var/ibm/biginsights/hadoop/logs and find the specific job folder and the right attempt folder, is it possible to see the log of an application directly from the web console ?

Thanks in advance
Updated on 2012-10-30T18:01:16Z at 2012-10-30T18:01:16Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    603 Posts
    ACCEPTED ANSWER

    Re: Use HBase under BIgInsights

    ‏2012-10-24T19:05:27Z  in response to SystemAdmin
    Hi JacopoF,

    Could you provide me the Biginsights version that you are using?

    Thanks,
    Harish
  • SystemAdmin
    SystemAdmin
    603 Posts
    ACCEPTED ANSWER

    Re: Use HBase under BIgInsights

    ‏2012-10-25T07:32:19Z  in response to SystemAdmin
    Yes, it's BigInsights Enterprise 1.4.0.0, running ad a single-node under CentOS 6.3 x86_64 inside a VirtualBox instance.
    • SystemAdmin
      SystemAdmin
      603 Posts
      ACCEPTED ANSWER

      Re: Use HBase under BIgInsights

      ‏2012-10-25T18:27:45Z  in response to SystemAdmin
      Can you try the following and let me know

      Check the status of hbase by running /opt/ibm/biginsights/bin/status.sh hbase -- This will give you the status of the hbase Master and Region Servers. If any of the servers are down, run the following command /opt/ibm/biginsights/bin/stop.sh hbase and then /opt/ibm/biginsights/bin/start.sh hbase
      • SystemAdmin
        SystemAdmin
        603 Posts
        ACCEPTED ANSWER

        Re: Use HBase under BIgInsights

        ‏2012-10-26T07:56:42Z  in response to SystemAdmin
        The output is:

        INFO Progress - Status hbase
        INFO @localhost.localdomain - hbase-master(active) started
        INFO @localhost.localdomain - hbase-regionserver started
        INFO Progress - 100%
        INFO DeployManager - Status; SUCCEEDED components: hbase]; FAILED components: [

        However, I restarted with stop-al.sh and start-all.sh and even restarted the whole server (it's just a VM used for tests), but it still happens.
        • SystemAdmin
          SystemAdmin
          603 Posts
          ACCEPTED ANSWER

          Re: Use HBase under BIgInsights

          ‏2012-10-26T18:27:40Z  in response to SystemAdmin
          Thanks Jacopo. If you have your application deployed on the Biginsights console, can you try the following.

          1) Open your application on the web console
          2) Click on the 'Configure' icon the top right corner. A configure dialog box will open.
          3) Check if you have any path listed under RuntimeDependencies section. If nothing is listed, add the path to the hbase jar (hdfs path) there and try to run the app.
          • SystemAdmin
            SystemAdmin
            603 Posts
            ACCEPTED ANSWER

            Re: Use HBase under BIgInsights

            ‏2012-10-29T09:39:00Z  in response to SystemAdmin
            Thanks, now it almost works!
            Is there a way to automatically include the hbase jar insted of add it from the console each time?
            • SystemAdmin
              SystemAdmin
              603 Posts
              ACCEPTED ANSWER

              Re: Use HBase under BIgInsights

              ‏2012-10-29T17:38:54Z  in response to SystemAdmin
              You need not set the RuntimeDependencies each time you run the app. It is enough to set it once after you deploy the app.

              Setting RuntimeDependencies can be done by either adding the path using the console UI or the other way is, through the application.xml that is part of the application bundle (zip file)
              Inside the application.xml, you can set the RuntimeDependencies by including the path (comma separated) inside the <runtimeDependencies> tag as follows

              <runtimeDependencies>/path/to/jar/on/hdfs, /path/to/jar/on/hdfs</runtimeDependencies>.

              Hope this helps.
            • SystemAdmin
              SystemAdmin
              603 Posts
              ACCEPTED ANSWER

              Re: Use HBase under BIgInsights

              ‏2012-10-29T17:57:42Z  in response to SystemAdmin
              The following is an sample application.xml

              <application-template xmlns="http://biginsights.ibm.com/application" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
              <name>HBase</name>
              <description>The HBase app exports the table to csv format</description>
              <properties>
              <property description="Enter the table name." isInputPath="false" isOutputPath="false" isRequired="true" label="Table Name" name="tableName" paramtype="STRING" uitype="textfield"/>
              <property description="Enter the row key." isInputPath="false" isOutputPath="false" isRequired="true" label="Row Key" name="rowKey" paramtype="STRING" uitype="textfield"/>
              <property description="Enter the number of records to fetch." isInputPath="false" isOutputPath="false" isRequired="true" label="Number of records to fetch" name="numRecords" paramtype="STRING" uitype="textfield"/>
              <property defaultValue="csv" description="Output format of the data - csv, json" isInputPath="false" isOutputPath="false" isRequired="true" label="Output format" name="outputFormat" paramtype="LIST" uitype="textfield" validValues="csv, json"/>
              <property description="Output directory where the results will be saved." isDataset="true" isInputPath="false" isOutputPath="true" isRequired="true" label="Output Directory" name="outputDir" paramtype="DIRECTORYPATH" uitype="textfield"/>
              </properties>
              <assets>
              <asset id="HBase" type="WORKFLOW"/>
              </assets>
              <imagePath>HBase_L.png</imagePath>
              <categories>Database,Export</categories>
              <runtimeDependencies>/biginsights/oozie/sharedLibraries/hbase</runtimeDependencies>
              </application-template>

              Note: The tags are case sensitive.