Topic
  • 5 replies
  • Latest Post - ‏2015-04-08T17:19:14Z by mrmadira
MattRiemer
MattRiemer
2 Posts

Pinned topic Jaql JVM Heap Size Issue

‏2014-08-07T20:01:16Z |

I am using the dbms::jdbc module in Jaql for BigInsights 3.0. I am trying to pull a large table from a DBMS connection with code that works for smaller tables (I also have a requirement that I cannot use the partitioned query functionality). However, I get the following error:

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2014/08/07 15:54:23 - please wait.
JVMDUMP032I JVM requested Heap dump using '/opt/ibm/biginsights/jaql/bin/heapdump.20140807.155423.3658788.0001.phd' in response to an event
JVMDUMP010I Heap dump written to /opt/ibm/biginsights/jaql/bin/heapdump.20140807.155423.3658788.0001.phd
JVMDUMP032I JVM requested Java dump using '/opt/ibm/biginsights/jaql/bin/javacore.20140807.155423.3658788.0002.txt' in response to an event
JVMDUMP010I Java dump written to /opt/ibm/biginsights/jaql/bin/javacore.20140807.155423.3658788.0002.txt
JVMDUMP032I JVM requested Snap dump using '/opt/ibm/biginsights/jaql/bin/Snap.20140807.155423.3658788.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /opt/ibm/biginsights/jaql/bin/Snap.20140807.155423.3658788.0003.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
encountered an exception during the evaluation of a statement
java.lang.OutOfMemoryError: Java heap space
 

I have increased the heap size of map and reduce jobs and experienced no change, so I presume this Java heap space issue has to do with the Jaql client itself. Does anyone know if there is a way that I can increase the heap space that the Jaql client has access to when acting in a standalone capacity? 

Thank you in advance for the help!

  • mrmadira
    mrmadira
    26 Posts

    Re: Jaql JVM Heap Size Issue

    ‏2014-08-12T05:45:24Z  

    You can set the values for jaql process'  java heap settings Xmx using the command below and then run the jaqlshell

    export JAQL_JAVA_OPTS="-Xmx2g"

    Updated on 2014-09-03T05:37:35Z at 2014-09-03T05:37:35Z by mrmadira
  • MattRiemer
    MattRiemer
    2 Posts

    Re: Jaql JVM Heap Size Issue

    ‏2014-08-13T13:44:50Z  
    • mrmadira
    • ‏2014-08-12T05:45:24Z

    You can set the values for jaql process'  java heap settings Xmx using the command below and then run the jaqlshell

    export JAQL_JAVA_OPTS="-Xmx2g"

    Thank you for your suggestion! Unfortunately, I don't think this is working for my BigInsights (version 3.0). I used this command before opening Jaqlshell and still got an error about the JVM running out of memory (despite providing much more than enough memory in the export command you pointed out).

    I also was wondering whether anyone knows of a config file where this setting can be implemented across the cluster, or of a command that Jaql can run to set this during a script. The reason I ask, is because I am running my jobs as an application through the Adhoc Jaql Query app, and not Jaqlshell. 

  • mrmadira
    mrmadira
    26 Posts

    Re: Jaql JVM Heap Size Issue

    ‏2014-08-18T18:06:22Z  

    Thank you for your suggestion! Unfortunately, I don't think this is working for my BigInsights (version 3.0). I used this command before opening Jaqlshell and still got an error about the JVM running out of memory (despite providing much more than enough memory in the export command you pointed out).

    I also was wondering whether anyone knows of a config file where this setting can be implemented across the cluster, or of a command that Jaql can run to set this during a script. The reason I ask, is because I am running my jobs as an application through the Adhoc Jaql Query app, and not Jaqlshell. 

    To complete this thread:

    Matt was able to do a hack by modifying the $JAQL_HOME/bin/jaql script and pass in the Xmx param explicitly.

    Adhoc JAQL app feature is being addressed separtely in the product

  • GregMc
    GregMc
    3 Posts

    Re: Jaql JVM Heap Size Issue

    ‏2015-04-06T23:34:35Z  
    • mrmadira
    • ‏2014-08-18T18:06:22Z

    To complete this thread:

    Matt was able to do a hack by modifying the $JAQL_HOME/bin/jaql script and pass in the Xmx param explicitly.

    Adhoc JAQL app feature is being addressed separtely in the product

    Can you please tell me how the Adhoc JAQL application feature is addressed in BigInsights 3.0 or point me to any information about it.  I am experiencing Out of Memory errors as well with my JAQL app that uses SystemT to extract large volumes of  data and JDBC to write that data to the database.

  • mrmadira
    mrmadira
    26 Posts

    Re: Jaql JVM Heap Size Issue

    ‏2015-04-08T17:19:14Z  
    • GregMc
    • ‏2015-04-06T23:34:35Z

    Can you please tell me how the Adhoc JAQL application feature is addressed in BigInsights 3.0 or point me to any information about it.  I am experiencing Out of Memory errors as well with my JAQL app that uses SystemT to extract large volumes of  data and JDBC to write that data to the database.

    https://developer.ibm.com/hadoop/blog/2014/10/09/handling-jaql-memory-exceptions-integrating-biginsights-netezza-using-jaql_submit-interface/

    Check this out, Greg.