Topic
8 replies Latest Post - ‏2014-04-17T11:09:49Z by A. Panayotov
AlexanderIuras
AlexanderIuras
2 Posts
ACCEPTED ANSWER

Pinned topic How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

‏2014-01-14T13:41:50Z |

Hello everyone,

we are migrating from sdk 1.5 and WAS 6.1 to sdk 1.7 and was 8.5.5.1 and getting the following Exception at initialisation phase of Application after the WAS is startetd:

[1/14/14 14:06:16:422 CET] 000000d7 SharedEJBRunt E   WSVR0068E: Attempt to start EnterpriseBean Stammdatenpflege#Depl_FrameworkNGServer-wastoplink.jar#MasterRemotehandler failed with exception: ja
va.lang.VerifyError: JVMVRFY012 stack shape inconsistent
; class=com/foobar/framework/abo/appserver/MasterRemotehandlerBean, method=getProtectedRecordsMap(Lcom/foobar/framework/abo/Searc
hInfo;)Ljava/util/Map;, pc=170
        at java.lang.J9VMInternals.verifyImpl(Native Method)
        at java.lang.J9VMInternals.verify(J9VMInternals.java:93)
        at java.lang.J9VMInternals.prepare(J9VMInternals.java:490)

If I understand correct, this Error means that the application code is compiled by a different java SDK Version than the one that is running it. (WAS)

What i'd like to ask is how should one proceed at solving such a Problem? Are there some best practices, or articles?

Should I pull some Java Cores and search there for more detailed information? Like exact version and name of the original java SDK. Like someone did here. 

Thanks.

  • A. Panayotov
    A. Panayotov
    6 Posts
    ACCEPTED ANSWER

    Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

    ‏2014-01-14T15:09:14Z  in response to AlexanderIuras

    Hello, Alexander

    I experienced the same error a while ago when I was trying to upgrade from IBM JRE 1.7.0 SR3 to 1.7.0 SR5. The research about this very error has been going ever since. So far I have found the following things:

    1) in SR5, IBM introduced an improved bytecode verification, which obviously doesn't like bytecode that has been compiled for/from a different version.
    2)  if the bytecode was complied for/from the same version and adjusted afterwards (with tools such as AspectJWeaver and similar), which is my very case, the IBM JVM doesn't like that as well.

    So, in case you are not using any bytecode modifications, you might get lucky. Try compiling the affected classes with a newer version of the JDK.
    If you are indeed using tools like AspectJWeaver, can't help you. I have opened a bug report in the Eclipse Foundation bug tracking system and I am waiting for a response.

    Best regards,

    A. Panayotov 

    • AlexanderIuras
      AlexanderIuras
      2 Posts
      ACCEPTED ANSWER

      Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

      ‏2014-01-16T10:20:27Z  in response to A. Panayotov

      Hello Mr. Panayotov,

      thank you for answering.

      1. No, we are not using AspectJ or similar tools.

      2. I'm about to find out the exact Version of the Build SDK, it's in anothers Team responsibility.

      But you've given me a hint, in December we upgraded our WAS SDK to 7.0.5 and it could be that the Build Team is still on the 7.0.1 Release. I'll check that.

      What I'm still wondering is if IBM JVMs have something similar like "-XX:-UseSplitVerifier" in Oracle JVMs? This tells the JVM to use the old TypeVerifier, here it is explained. Till now i can't find it in the IBM JVM Info Center so it looks like it's not implemented in IBM JVMs. :(

      I think if IBM would offer such a parameter it could solve our Problems "the easy way". (-;

       

       

      • A. Panayotov
        A. Panayotov
        6 Posts
        ACCEPTED ANSWER

        Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

        ‏2014-01-16T11:14:09Z  in response to AlexanderIuras

        Alexander,

        The JVM option -XX:-UseSplitVerifier was suggested as a possible workaround in the AspectJ community forums. However, so far it hasn't worked for me. I haven't dug any further, but it might be that the IBM JVM doesn't accept such an option or something else is messed up. I can say for sure that bytecode compiled with any (so far) version of Java 7 (be it the IBM JDK, the Oracle one, etc) is accepted by the IBM JVM (SR5+). We are using Oracle JDK 1.7.0_02 (pretty much the first stable version of the Java 7 JDK) on the build machines and yet if the AspectJ is disabled, things seemed to be working as supposed to.

         

        Edit #1: you can always try to add the "-XX:-UseSplitVerifier" to the JAVA_OPTS options string  and see if it changes anything.

        Updated on 2014-01-16T11:14:56Z at 2014-01-16T11:14:56Z by A. Panayotov
  • A. Panayotov
    A. Panayotov
    6 Posts
    ACCEPTED ANSWER

    Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

    ‏2014-01-24T09:15:40Z  in response to AlexanderIuras

    Alexander,

    Even though the case is closed, I would like to post my latest conclusions based on even further investigation.

    It appears that the JVM in IBM JRE/JDK 1.7.0 SR5 has been messed up. Updating to SR6 seems to have fixed our issues and the JVMVRFY012 error is no longer reproducible, although this applies to the case with AspectJ bytecode weaving. Consider trying SR6 - it might help you solve your issue as well.

     

    • tonych@ng
      tonych@ng
      2 Posts
      ACCEPTED ANSWER

      Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

      ‏2014-04-16T06:56:52Z  in response to A. Panayotov

      Dear Panayotov,

      I have the same case in my application nowaday when upgrade from sdk 1.5 and WAS 6.1 to sdk 1.7 and was 8.5.5.1. Though upgrade to the latest IBM JRE/JDK 1.7.0 SR6 FP1, the issue still the same. So here would you mind to give me more advise?  And by the way, a strange and dummy solution is that if I move some of simple code within the function where the verify error occur, it seems work, but this is not a good solution, coz nobody can tell how may how much such case in the app.

      • A. Panayotov
        A. Panayotov
        6 Posts
        ACCEPTED ANSWER

        Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

        ‏2014-04-16T08:19:10Z  in response to tonych@ng

        Hello, tony_cheng,

        To begin with, I would like to stress that I am not an IBM employee, nor am I related to IBM in any way, besides being a user of some of their products.

        OnTopic: The workaround you mentioned is definitely not a good one, especially in large applications.  This is because the inconsistency issue is, well, nasty as hell. You have seen it somewhere, but that does not mean it won't show up somewhere else.

        Anyways, some of the possible reasons for this are:
        1) Missing/corrupted/etc IBM JRE/JDK 1.7.0 SR6 jars in your project. taken from this articleProblems building application with Java 7 using WAS 8.0.0.4 jar files

        2) Make sure that all the bytecode has been compiled using sdk7.

        3) Are you using any tools that alter the bytecode @ runtime/build time? (Such tools are AspectJ, CA Introscope, etc) If so, that is most likely to be your issue.

        4) In case you are using Java Cryptography Extension, make sure you are using the appropriate version. According to the first article I mentioned, this might be causing the issue as well.

        Also, providing some stracktrace would be of much help.
        Waiting for more info, and if I find anything else, I will be updating the thread.

        BR,
        A. Panayotov

        Updated on 2014-04-16T08:20:54Z at 2014-04-16T08:20:54Z by A. Panayotov
        • tonych@ng
          tonych@ng
          2 Posts
          ACCEPTED ANSWER

          Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

          ‏2014-04-17T06:18:43Z  in response to A. Panayotov

          Dear Panayotov,

          Thanks for your quick reponse and patience answers, for your advice,

          1)My project build path/compiler is using the same IBM java7 within WAS8.5.5.1, so suppose no missing jars in the project.

          2)Except for those 3rt party jars and some jars development by other apps with lower jdk version, all sources are recompiled with the ibm java7 sr6. including the source where this varify error occur. I think no need to recompile others jars, right?

          3)My case just use the Rational Application Deveoper 8.5 to compile and the deploy the app.

          4) No JCE is using.

          and now another dummy solution is that,I am aware there is a local char variable is not initialized when declaration, for example:

                   char sampeChar ;

          when I change the code with initial value as below,

                    char sampeChar =0;

          the class verify pass and all the functions work well. What a dummy change! but it's not a good solution. I don't how many types of such code need to be changed during my Java7 migration. 

          So go back to the problem, Is it possible to close the IBM JVM7 bytecode verification or use the old verifier instead?

          Anyways, I try to setup the JVM parameter of "-XX:-UseSplitVerifier" in WAS8.5.5.1, but it make the same result. so suppose the parameter of "-XX:-UseSplitVerifier" is only supported by Oracle Standard JVM, am I right? And I also try the "-Xverify:none", but this parameter setting always disappear when I reboot the WAS to make it effective.

          Updated on 2014-04-17T06:19:52Z at 2014-04-17T06:19:52Z by tonych@ng
          • A. Panayotov
            A. Panayotov
            6 Posts
            ACCEPTED ANSWER

            Re: How to solve a java.lang.VerifyError JVMVRFY012 stack shape inconsistent Problem?

            ‏2014-04-17T11:09:49Z  in response to tonych@ng

            There should be a list of the compatible 3rd party libraries for WAS8.5.5.1. Using an incompatible 3rd party library is a possible cause of the issue. Also, if you are using jars compiled with lower JDK version (for example 1.6), you should rebuild them with JDK 1.7 using the compatibility switch (for example, if you are using ant for building, the switch is <javac (other attributes) source="1.6" target="1.6" (more attributes)  /> ). Unfortunately, I am not familiar with RAS 8.5 (or any RAS versions) so I have no knowledge there.

            The parameter "-XX: -UseSplitVerifier", as far as I know, is supported in the IBM J9VM. Also, disabling the verifier was an option prior to IBM JRE/JDK 1.7.0 SR3 (or even earlier versions). Now, the verifier can not be disabled.

            Regarding the dummy fix, in most IDEs, if a variable is not initialized, it results in a build-time error/warning which would rather cause the build to fail.

            Give it a try with rebuilding all the jars with jdk7 as described. If it still fails, perhaps you could post a stacktrace with omitted class name literals for confidentiality reasons.

            HTH,
            A. Panayotov