Topic
  • 8 replies
  • Latest Post - ‏2013-06-17T08:33:56Z by CarpLi
sakus
sakus
5 Posts

Pinned topic Machine Data Accelerator installation problem

‏2013-06-03T11:57:19Z |

 

Hi,

We tried to install Machine Data Accelerator on BigInsights 2.0, but received the following errors identified in the debug log below. Any ideas what went wrong?

.

.

.

 

lax.nl.message.vm.not.loaded=The installer either could not find a Java VM, or the Java VM on this system is too old. The installer requires Java 1.1.5 or later. It can be downloaded from http://java.sun.com/products/jdk/1.1/jre/lax.nl.message.vm.not.loaded=The installer either could not find a Java VM, or the Java VM on this system is too old. The installer requires Java 1.1.5 or later. It can be downloaded from http://java.sun.com/products/jdk/1.1/jre/

.

.

.

 

Copying the MDAConfig.json file to /user/applications/MDA
 
Copy Files to DFS stderrMsg: JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
 
 

Br,
Saku

 

Attachments

Updated on 2013-06-03T12:17:26Z at 2013-06-03T12:17:26Z by sakus
  • CarpLi
    CarpLi
    3 Posts
    ACCEPTED ANSWER

    Re: Machine Data Accelerator installation problem

    ‏2013-06-17T08:33:56Z  
    • sakus
    • ‏2013-06-17T08:30:57Z

    Hi,

    The environment is now more stable. We were able to identify some parameters changes which improved the performance. Primarily these were at

    sysctl.conf

    security/limits.conf

    config files.

     

    Thanks,

    Saku

    Oh, good to know, so it seems the issue already got resolved and you can install MDA now, right? Just let us know if still got any issue there.

  • JesusAlvarez
    JesusAlvarez
    4 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-03T20:59:19Z  

    Hi Saku,

    From what I see there is 2 possible places this went wrong,

    1. Insufficient memory settings for the JVM. Causing the VM to not start properly. http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp?topic=%2Fcom.ibm.java.messages%2Fdiag%2Fappendixes%2Fmessages%2Fj9gc%2Fmessage_jvmj9gc070.html When the installer is run, it's reporting: Free Memory: 824 kB Total Memory: 4096 kB. 

    2. Run "java -version" and ensure that Java 1.6 is installer on your cluster. If not, manually update prior to running the installation. 

  • sakus
    sakus
    5 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-04T07:04:17Z  

    Hi Saku,

    From what I see there is 2 possible places this went wrong,

    1. Insufficient memory settings for the JVM. Causing the VM to not start properly. http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp?topic=%2Fcom.ibm.java.messages%2Fdiag%2Fappendixes%2Fmessages%2Fj9gc%2Fmessage_jvmj9gc070.html When the installer is run, it's reporting: Free Memory: 824 kB Total Memory: 4096 kB. 

    2. Run "java -version" and ensure that Java 1.6 is installer on your cluster. If not, manually update prior to running the installation. 

    Thanks Jesus for your reply. The Java versions should be inline with the requirements.

     

    [root@mgm01 ~]# /opt/ibm/biginsights/jdk/bin/ java -version

    java version "1.6.0"

    Java(TM) SE Runtime Environment (build pxa6460sr11-20120806_01(SR11))

    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr11-20120801_118201 (JIT enabled, AOT enabled)

    J9VM - 20120801_118201

    JIT  - r9_20120608_24176ifx1

    GC   - 20120516_AA)

    JCL  - 20120713_01

    [root@mgm01 ~]# /opt/ibm/accelerators/MDA/jre/ bin/java -version

    java version "1.6.0"

    Java(TM) SE Runtime Environment (build pxa6460sr10-20111208_01(SR10))

    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr10-20111207_96808 (JIT enabled, AOT enabled)

    J9VM - 20111207_096808

    JIT  - r9_20111107_21307ifx1

    GC   - 20110519_AA)

    JCL  - 20111104_02

    There should also be enough memory.

     

    Could this be the reason? Install Anywhere/LauchAnyware configuration found in the same log file.

     

    lax.nl.java.option.java.heap. size.initial=16777216

    lax.nl.java.option.java.heap. size.max=50331648 (~48 mbytes)

  • JesusAlvarez
    JesusAlvarez
    4 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-06T17:26:12Z  
    • sakus
    • ‏2013-06-04T07:04:17Z

    Thanks Jesus for your reply. The Java versions should be inline with the requirements.

     

    [root@mgm01 ~]# /opt/ibm/biginsights/jdk/bin/ java -version

    java version "1.6.0"

    Java(TM) SE Runtime Environment (build pxa6460sr11-20120806_01(SR11))

    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr11-20120801_118201 (JIT enabled, AOT enabled)

    J9VM - 20120801_118201

    JIT  - r9_20120608_24176ifx1

    GC   - 20120516_AA)

    JCL  - 20120713_01

    [root@mgm01 ~]# /opt/ibm/accelerators/MDA/jre/ bin/java -version

    java version "1.6.0"

    Java(TM) SE Runtime Environment (build pxa6460sr10-20111208_01(SR10))

    IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr10-20111207_96808 (JIT enabled, AOT enabled)

    J9VM - 20111207_096808

    JIT  - r9_20111107_21307ifx1

    GC   - 20110519_AA)

    JCL  - 20111104_02

    There should also be enough memory.

     

    Could this be the reason? Install Anywhere/LauchAnyware configuration found in the same log file.

     

    lax.nl.java.option.java.heap. size.initial=16777216

    lax.nl.java.option.java.heap. size.max=50331648 (~48 mbytes)

    It could be.. though the installer heap is pre-set into the installer and should not be an issue.

    One thing that I noticed is that it fails exactly when it tries to upload a file to DFS (and the java error comes from the garbage collector failing to start). 

    I was able to reproduce the error by having hadoop stopped -- Can you ensure that the console and hadoop services are running? These error logs were significantly improved for clarity in the coming release.

    (as biadmin)
    cd $BIGINSIGHTS_HOME/bin
    start.sh hadoop console

     

    Regards,

    Jesus

  • sakus
    sakus
    5 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-13T16:55:37Z  

    It could be.. though the installer heap is pre-set into the installer and should not be an issue.

    One thing that I noticed is that it fails exactly when it tries to upload a file to DFS (and the java error comes from the garbage collector failing to start). 

    I was able to reproduce the error by having hadoop stopped -- Can you ensure that the console and hadoop services are running? These error logs were significantly improved for clarity in the coming release.

    (as biadmin)
    cd $BIGINSIGHTS_HOME/bin
    start.sh hadoop console

     

    Regards,

    Jesus

    Hi Jesus,

    It seams that the error message is not related to MDA. We get the same error message also even though MDA is not installed. For example if we try to import multiple files to HDFS using sftp, the system gets stuck. If we try to stop the services (./stop-all.sh), we get the same error message:

     

    JVMJ9GC070E Failed to startup the Garbage Collector
    Could not create the Java virtual machine.

    Getting really frustrated with this. Sometimes the services stay up and running for a while, sometimes just few services stay up and running for a while, sometimes it seams that no services want to reply. We get also message during service startup that the name node is in safemode. After stopping all services name node seams to come back for just to go to safe mode again after a minor action taken in import or mapreduce job. 

    Out config includes one physical management node and one physical data node. The data node disks are shown from a centralized storage and there are all together 10 luns.

    Br,

    Saku

  • JesusAlvarez
    JesusAlvarez
    4 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-13T18:55:46Z  
    • sakus
    • ‏2013-06-13T16:55:37Z

    Hi Jesus,

    It seams that the error message is not related to MDA. We get the same error message also even though MDA is not installed. For example if we try to import multiple files to HDFS using sftp, the system gets stuck. If we try to stop the services (./stop-all.sh), we get the same error message:

     

    JVMJ9GC070E Failed to startup the Garbage Collector
    Could not create the Java virtual machine.

    Getting really frustrated with this. Sometimes the services stay up and running for a while, sometimes just few services stay up and running for a while, sometimes it seams that no services want to reply. We get also message during service startup that the name node is in safemode. After stopping all services name node seams to come back for just to go to safe mode again after a minor action taken in import or mapreduce job. 

    Out config includes one physical management node and one physical data node. The data node disks are shown from a centralized storage and there are all together 10 luns.

    Br,

    Saku

    Hi Saku,

    So this does seem to be a Java native memory size issue (Error JVMJ9GC070E)

    I'm working directly on the MDA accelerator and Accelerator installers- so let me contact someone from the platform side to join this thread and provide a solution for this. 

     

    Regards,

    Jesus

  • CarpLi
    CarpLi
    3 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-17T08:22:52Z  
    • sakus
    • ‏2013-06-13T16:55:37Z

    Hi Jesus,

    It seams that the error message is not related to MDA. We get the same error message also even though MDA is not installed. For example if we try to import multiple files to HDFS using sftp, the system gets stuck. If we try to stop the services (./stop-all.sh), we get the same error message:

     

    JVMJ9GC070E Failed to startup the Garbage Collector
    Could not create the Java virtual machine.

    Getting really frustrated with this. Sometimes the services stay up and running for a while, sometimes just few services stay up and running for a while, sometimes it seams that no services want to reply. We get also message during service startup that the name node is in safemode. After stopping all services name node seams to come back for just to go to safe mode again after a minor action taken in import or mapreduce job. 

    Out config includes one physical management node and one physical data node. The data node disks are shown from a centralized storage and there are all together 10 luns.

    Br,

    Saku

    Hi Saku,

    Please give me below information for further investigation:

    1) Log on BigInsights management node

    2) Run "listnode.sh hadoop", capture and paste output here

    3) Run "status.sh hadoop", capture and paste output here

    4) Run "hadoop fs -put $BIGINSIGHTS_HOME/hbase/conf/* /tmp", capture and paste the output here; then collect log files in $BIGINSIGHTS_VAR/hadoop/logs on all hadoop nodes(Only need to collect files whose names end with ".log") and attach here

    5) Use "top" to get the top 3 pid which cost most memory, then use "ps" to get the process details of these pids, paste the process information here

     

    Regards,

    Yu

  • sakus
    sakus
    5 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-17T08:30:57Z  
    • CarpLi
    • ‏2013-06-17T08:22:52Z

    Hi Saku,

    Please give me below information for further investigation:

    1) Log on BigInsights management node

    2) Run "listnode.sh hadoop", capture and paste output here

    3) Run "status.sh hadoop", capture and paste output here

    4) Run "hadoop fs -put $BIGINSIGHTS_HOME/hbase/conf/* /tmp", capture and paste the output here; then collect log files in $BIGINSIGHTS_VAR/hadoop/logs on all hadoop nodes(Only need to collect files whose names end with ".log") and attach here

    5) Use "top" to get the top 3 pid which cost most memory, then use "ps" to get the process details of these pids, paste the process information here

     

    Regards,

    Yu

    Hi,

    The environment is now more stable. We were able to identify some parameters changes which improved the performance. Primarily these were at

    sysctl.conf

    security/limits.conf

    config files.

     

    Thanks,

    Saku

  • CarpLi
    CarpLi
    3 Posts

    Re: Machine Data Accelerator installation problem

    ‏2013-06-17T08:33:56Z  
    • sakus
    • ‏2013-06-17T08:30:57Z

    Hi,

    The environment is now more stable. We were able to identify some parameters changes which improved the performance. Primarily these were at

    sysctl.conf

    security/limits.conf

    config files.

     

    Thanks,

    Saku

    Oh, good to know, so it seems the issue already got resolved and you can install MDA now, right? Just let us know if still got any issue there.