Topic
8 replies Latest Post - ‏2013-06-17T08:33:56Z by CarpLi
sakus
sakus
5 Posts
ACCEPTED ANSWER

Pinned topic Machine Data Accelerator installation problem

‏2013-06-03T11:57:19Z |

 

Hi,

We tried to install Machine Data Accelerator on BigInsights 2.0, but received the following errors identified in the debug log below. Any ideas what went wrong?

.

.

.

 

lax.nl.message.vm.not.loaded=The installer either could not find a Java VM, or the Java VM on this system is too old. The installer requires Java 1.1.5 or later. It can be downloaded from http://java.sun.com/products/jdk/1.1/jre/lax.nl.message.vm.not.loaded=The installer either could not find a Java VM, or the Java VM on this system is too old. The installer requires Java 1.1.5 or later. It can be downloaded from http://java.sun.com/products/jdk/1.1/jre/

.

.

.

 

Copying the MDAConfig.json file to /user/applications/MDA
 
Copy Files to DFS stderrMsg: JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
JVMJ9GC070E Failed to startup the Garbage Collector
Could not create the Java virtual machine.
 
 

Br,
Saku

 

Attachments

Updated on 2013-06-03T12:17:26Z at 2013-06-03T12:17:26Z by sakus
  • JesusAlvarez
    JesusAlvarez
    4 Posts
    ACCEPTED ANSWER

    Re: Machine Data Accelerator installation problem

    ‏2013-06-03T20:59:19Z  in response to sakus

    Hi Saku,

    From what I see there is 2 possible places this went wrong,

    1. Insufficient memory settings for the JVM. Causing the VM to not start properly. http://publib.boulder.ibm.com/infocenter/java7sdk/v7r0/index.jsp?topic=%2Fcom.ibm.java.messages%2Fdiag%2Fappendixes%2Fmessages%2Fj9gc%2Fmessage_jvmj9gc070.html When the installer is run, it's reporting: Free Memory: 824 kB Total Memory: 4096 kB. 

    2. Run "java -version" and ensure that Java 1.6 is installer on your cluster. If not, manually update prior to running the installation. 

    • sakus
      sakus
      5 Posts
      ACCEPTED ANSWER

      Re: Machine Data Accelerator installation problem

      ‏2013-06-04T07:04:17Z  in response to JesusAlvarez

      Thanks Jesus for your reply. The Java versions should be inline with the requirements.

       

      [root@mgm01 ~]# /opt/ibm/biginsights/jdk/bin/ java -version

      java version "1.6.0"

      Java(TM) SE Runtime Environment (build pxa6460sr11-20120806_01(SR11))

      IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr11-20120801_118201 (JIT enabled, AOT enabled)

      J9VM - 20120801_118201

      JIT  - r9_20120608_24176ifx1

      GC   - 20120516_AA)

      JCL  - 20120713_01

      [root@mgm01 ~]# /opt/ibm/accelerators/MDA/jre/ bin/java -version

      java version "1.6.0"

      Java(TM) SE Runtime Environment (build pxa6460sr10-20111208_01(SR10))

      IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr10-20111207_96808 (JIT enabled, AOT enabled)

      J9VM - 20111207_096808

      JIT  - r9_20111107_21307ifx1

      GC   - 20110519_AA)

      JCL  - 20111104_02

      There should also be enough memory.

       

      Could this be the reason? Install Anywhere/LauchAnyware configuration found in the same log file.

       

      lax.nl.java.option.java.heap. size.initial=16777216

      lax.nl.java.option.java.heap. size.max=50331648 (~48 mbytes)

      • JesusAlvarez
        JesusAlvarez
        4 Posts
        ACCEPTED ANSWER

        Re: Machine Data Accelerator installation problem

        ‏2013-06-06T17:26:12Z  in response to sakus

        It could be.. though the installer heap is pre-set into the installer and should not be an issue.

        One thing that I noticed is that it fails exactly when it tries to upload a file to DFS (and the java error comes from the garbage collector failing to start). 

        I was able to reproduce the error by having hadoop stopped -- Can you ensure that the console and hadoop services are running? These error logs were significantly improved for clarity in the coming release.

        (as biadmin)
        cd $BIGINSIGHTS_HOME/bin
        start.sh hadoop console

         

        Regards,

        Jesus

        • sakus
          sakus
          5 Posts
          ACCEPTED ANSWER

          Re: Machine Data Accelerator installation problem

          ‏2013-06-13T16:55:37Z  in response to JesusAlvarez

          Hi Jesus,

          It seams that the error message is not related to MDA. We get the same error message also even though MDA is not installed. For example if we try to import multiple files to HDFS using sftp, the system gets stuck. If we try to stop the services (./stop-all.sh), we get the same error message:

           

          JVMJ9GC070E Failed to startup the Garbage Collector
          Could not create the Java virtual machine.

          Getting really frustrated with this. Sometimes the services stay up and running for a while, sometimes just few services stay up and running for a while, sometimes it seams that no services want to reply. We get also message during service startup that the name node is in safemode. After stopping all services name node seams to come back for just to go to safe mode again after a minor action taken in import or mapreduce job. 

          Out config includes one physical management node and one physical data node. The data node disks are shown from a centralized storage and there are all together 10 luns.

          Br,

          Saku

          • JesusAlvarez
            JesusAlvarez
            4 Posts
            ACCEPTED ANSWER

            Re: Machine Data Accelerator installation problem

            ‏2013-06-13T18:55:46Z  in response to sakus

            Hi Saku,

            So this does seem to be a Java native memory size issue (Error JVMJ9GC070E)

            I'm working directly on the MDA accelerator and Accelerator installers- so let me contact someone from the platform side to join this thread and provide a solution for this. 

             

            Regards,

            Jesus

          • CarpLi
            CarpLi
            3 Posts
            ACCEPTED ANSWER

            Re: Machine Data Accelerator installation problem

            ‏2013-06-17T08:22:52Z  in response to sakus

            Hi Saku,

            Please give me below information for further investigation:

            1) Log on BigInsights management node

            2) Run "listnode.sh hadoop", capture and paste output here

            3) Run "status.sh hadoop", capture and paste output here

            4) Run "hadoop fs -put $BIGINSIGHTS_HOME/hbase/conf/* /tmp", capture and paste the output here; then collect log files in $BIGINSIGHTS_VAR/hadoop/logs on all hadoop nodes(Only need to collect files whose names end with ".log") and attach here

            5) Use "top" to get the top 3 pid which cost most memory, then use "ps" to get the process details of these pids, paste the process information here

             

            Regards,

            Yu

            • sakus
              sakus
              5 Posts
              ACCEPTED ANSWER

              Re: Machine Data Accelerator installation problem

              ‏2013-06-17T08:30:57Z  in response to CarpLi

              Hi,

              The environment is now more stable. We were able to identify some parameters changes which improved the performance. Primarily these were at

              sysctl.conf

              security/limits.conf

              config files.

               

              Thanks,

              Saku

              • CarpLi
                CarpLi
                3 Posts
                ACCEPTED ANSWER

                Re: Machine Data Accelerator installation problem

                ‏2013-06-17T08:33:56Z  in response to sakus

                Oh, good to know, so it seems the issue already got resolved and you can install MDA now, right? Just let us know if still got any issue there.