Topic
14 replies Latest Post - ‏2008-01-08T06:21:23Z by SystemAdmin
SystemAdmin
SystemAdmin
84 Posts
ACCEPTED ANSWER

Pinned topic Support Dynamic Bytecode Enhancement?

‏2007-12-24T17:56:05Z |
This was written initially for JDK 1.3 and now requires 1.4. Is there work on developing a javaagent to dynamically instrument the classes while they are loaded?

E.g. add the following jvm argument: -javaagent:ConTest.jar so that rather than needing a plugin for eclipse or a build script to instrument before running tests, we can simply just run the VM with the argument and get dynamic bytecode loading?

If not, can I get a hold of the source code so I can write (and then contribute back to the tool) such a class?
Updated on 2008-01-08T06:21:23Z at 2008-01-08T06:21:23Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    84 Posts
    ACCEPTED ANSWER

    Re: Support Dynamic Bytecode Enhancement?

    ‏2007-12-26T14:39:35Z  in response to SystemAdmin
    Brett,

    Yes, we are working on enabling instrumentation at class-load. Schedule is unclear at the moment, probably will take a few months.

    Since you are willing to do it yourself - unfortunately we can give the sources, but in the coming days I'll try to upload to alphaWorks a new version with public API for stream instrumentation - this should enable you to do class-load instrumentation.

    Regards,
    Yarden Nir-Buchbinder
    Software Testing, Verification and Review Methodologies
    IBM Haifa Research Lab
    • SystemAdmin
      SystemAdmin
      84 Posts
      ACCEPTED ANSWER

      Re: Support Dynamic Bytecode Enhancement?

      ‏2007-12-28T10:41:16Z  in response to SystemAdmin
      Yarden,

      I've got a generic Java agent jar written... (Apparently Java agents must reside in jars.) All I need to do is provide a class that implements the ClassFileTransformer interface. The interface has one method:

      byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException;

      So long as you don't need anything more than this (primarily the byte array), it should be pretty easy to dynamically inject the bytecode.
      Brett

      Message was edited by: Brett_Schuchert
      Updated on 2007-12-28T10:41:16Z at 2007-12-28T10:41:16Z by SystemAdmin
      • SystemAdmin
        SystemAdmin
        84 Posts
        ACCEPTED ANSWER

        Re: Support Dynamic Bytecode Enhancement?

        ‏2007-12-30T04:39:56Z  in response to SystemAdmin
        OK, so here's the "generic" ClassFileTransformer registrar: http://schuchert.wikispaces.com/JavaAgent

        I'm hoping I can take your API and create an adaptor for it. The adapter could use the following interface:

        package proposed.api.usage;

        public interface ThreadClassInstrumentor {
            byte[] instrument(ClassLoader loader, String className,
            Class<?> classBeingRedefined, byte[] classfileBuffer);
        }
        <hr />

        Assuming I can write a concrete implementation of this class that uses the API, then I will have a very simple ClassFileTransformer implementation:

        package proposed.api.usage;

        import java.lang.instrument.ClassFileTransformer;
        import java.lang.instrument.IllegalClassFormatException;
        import java.security.ProtectionDomain;

        public class ConTestDynamicClassTransformer implements ClassFileTransformer {
            private final ThreadClassInstrumentor instrumentor;

            public ConTestDynamicClassTransformer(ThreadClassInstrumentor instrumentor) {
                this.instrumentor = instrumentor;
            }

            public byte[] transform(ClassLoader loader, String className,
                    Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
                    byte[] classfileBuffer) throws IllegalClassFormatException {

                byte[] conTestInstrumentedClassfileBuffer = instrumentor.instrument(
                        loader, className, classBeingRedefined, classfileBuffer);

                return conTestInstrumentedClassfileBuffer;
            }
        }
        <hr />
        I've got a generic registrar in a jar that will make it easy to use with the -javaagenet VM argument - though what I've written so far does not allow passing in parameters to a constructor...I'll fix that in the next version if I know I can expect something like that.

        In lieu of of that, I could simply write a class more like this (forgo the use of the Adaptor pattern):

        package proposed.api.usage;

        import java.lang.instrument.ClassFileTransformer;
        import java.lang.instrument.IllegalClassFormatException;
        import java.security.ProtectionDomain;

        public class ConTestDynamicClassTransformer implements ClassFileTransformer {
            private final ContestApi instrumentor;

            public ConTestDynamicClassTransformer() {
                instrumentor = instantiateConTestApi();
            }

            public byte[] transform(ClassLoader loader, String className,
                    Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
                    byte[] classfileBuffer) throws IllegalClassFormatException {

                byte[] conTestInstrumentedClassfileBuffer = instrumentor.instrument(
                        loader, className, classBeingRedefined, classfileBuffer);

                return conTestInstrumentedClassfileBuffer;
            }

            private ContestApi instantiateConTestApi() {
                // to be implemented...
                return null;
            }
        }
        • SystemAdmin
          SystemAdmin
          84 Posts
          ACCEPTED ANSWER

          Re: Support Dynamic Bytecode Enhancement?

          ‏2007-12-31T08:04:19Z  in response to SystemAdmin
          I just successfully used dynamic instrumentation with ConTest. I wrote a non-thread-safe class:
          package example;

          public class ClassWithThreadingProblem {
              int commonValue;

              int nextValue() {
                  return ++commonValue;
              }

              int currentValue() {
             &nbsp&nbsp;   return commonValue;
              }
          }

          I then wrote a test class that would try and expose this class as a problem child:
          package example;

          import static org.junit.Assert.assertFalse;

          import org.junit.Test;

          public class ClassWithThreadingProblemTest {
              boolean failed;

              @Test
              public void twoThreadsShouldFailEventually() throws Exception {
                  final ClassWithThreadingProblem classWithThreadingProblem = new ClassWithThreadingProb lem();

                  Runnable runnable = new Runnable() {
                      public void run() {
                          long currentValue = classWithThreadingProblem.cu rrentValue();
                          long result = classWithThreadingProblem.nextValu e();
                          long expected = currentValue + 1;
                          if (expected != result)
                              failed = true;
                      }
                  };

                  for (int i = 0; i < 10000; ++i) {
                      Thread t1 = new Thread(runnable);
                      Thread t2 = new Thread(runnable);
                      t1.start();
                      t2.start();
                      t1.join();
                      t2.join();
                      assertFalse("Failed at increment: " + (i + 1),  failed);
                  }
              }
          }

          When I ran this without instrumentation, it typically passed - which is not what I wanted but it was what I expected. When I instrumented it with ConTest, it failed every time - and quickly. But to get my code instrumented, I did not use the command line, I wrote a Java Agent that essentially calls the command line on a class by class basis. All I need to do was add the following VM arguments:
          -javaagent:../JavaAgentRegistrar/Registrar.jar -Dschuchert.ClassFileTransformer=hacky.filebased.agent.ConTestInstrumentationMainBasedClassFileTransformer

          It's ugly, but effective.

          I'll post the work after I've had a little time to clean up the implementation.
          • SystemAdmin
            SystemAdmin
            84 Posts
            ACCEPTED ANSWER

            Re: Support Dynamic Bytecode Enhancement?

            ‏2007-12-31T08:32:42Z  in response to SystemAdmin
            Brett,

            That's exciting! Did you implement the instrumentation with the API of file instrumentation?

            The version with API for stream instrumentation is ready, and it should be a matter of days before the alphaWorks team uploads it. A version with classload instrumentation is pobably also nearer than I thought - the code is already there, what remains is some testing, and rewriting the user manual...
            • SystemAdmin
              SystemAdmin
              84 Posts
              ACCEPTED ANSWER

              Re: Support Dynamic Bytecode Enhancement?

              ‏2007-12-31T18:23:54Z  in response to SystemAdmin
              Yarden,

              I called the main method of the Instrumentation class (again, this still needs some refactoring):

              public byte[] transform(ClassLoader loader, String fullyQualifiedClassName,
                      Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
                      byte[] classfileBuffer) throws IllegalClassFormatException {

                  if (isClassWeShouldProcess(fullyQualifiedClassName)) {
                      try {
                          File fileWritten = FileUtilities.writeClassFile(tempDirectory, fullyQualifiedClassName, classfileBuffer);
                          
                          Instrument.main(new String[] { fileWritten.getAbsolutePath() });
                          
                          byte[] newClassBytes = FileUtilities.readGeneratedClassFile(fileWritten) ;

                          return newClassBytes;
                      } catch (Exception e) {
                          e.printStackTrace(System.err);
                      }
                  }

                  return null;
              }

              Now that I've done that, I can probably use the API for file instrumentation. I saw a class called OneClassInstrumentor but it has package level access, so I ignored it. Is that the class you're talking about?
              • SystemAdmin
                SystemAdmin
                84 Posts
                ACCEPTED ANSWER

                Re: Support Dynamic Bytecode Enhancement?

                ‏2008-01-01T06:27:05Z  in response to SystemAdmin
                No, OneClassInstrumentor is already too late down the chain. With the ConTest version now in site you can't do better than what you did (which is of course sad, because the "adaptor" you have to use involves you writing a file, ConTest reading it, ConTest writing a file, and you reading it). When the new version is on site, I'll update you here regarding the API to use.
  • SystemAdmin
    SystemAdmin
    84 Posts
    ACCEPTED ANSWER

    Re: Support Dynamic Bytecode Enhancement?

    ‏2008-01-07T06:34:07Z  in response to SystemAdmin
    Version 2.6.5.3 now in site has ClassStreamInstrumentor (and the required InstrumentationAction) which you should use in your agent.

    (Yes, as you spotted in the other thread, it also has a Premain, but it's in mid-development. Use it at your own risk...)

    I expect a version with class-load instrumentaiton would be ready in few weeks, possibly only few days.

    Yarden Nir-Buchbinder
    IBM Haifa Research Lab
    • SystemAdmin
      SystemAdmin
      84 Posts
      ACCEPTED ANSWER

      Re: Support Dynamic Bytecode Enhancement?

      ‏2008-01-07T10:16:12Z  in response to SystemAdmin
      Yarden,

      Nice work. Took about 10 minutes to:
      1. Rewrite the transform method to use this new class.
      2. Remove all code dealing with defining a temp directory my original version used.
      3. Remove all code dealing with deleting temporary file.


      It was very quick work because the ClassStreamInstrumentor was a snap to use and I just made the changes and verified nothing was broken by running my unit tests. I also checked that my smoke test, schuchert.agent.Main, also worked as expected.


      So the new class seems to work like a charm.


      I've updated the Registrar.jar on my wikisite (http://schuchert.wikispaces.com/JavaAgent and/or http://schuchert.wikispaces.com/DebuggingThreadedCodePart1), so people can experiment with it. I've also removed reference to the two removed properties. The only thing I have now is the ability to describe a filter using a :: separated list of regular expressions. ConTest has the ability to do the same, so I might remove that as well. I needed something easy, however, to make sure in my test that ConTest was not instrumented as well as java.* (some of the classes cause ConTest's instrumentation some problems - but then it wasn't designed to deal with those classes, so that's not a problem).


      I also updated the Eclipse workspace so people can give it a try.


      FYI, everything on that site is released under http://creativecommons.org/licenses/by-sa/2.5/. Not that you'll need to use any of it, but it's there if you want it.

      Brett

      Message was edited by: Brett_Schuchert - added links to find Registrar.jar, mentioned license of stuff on wikispaces site.
      Updated on 2008-01-07T10:16:12Z at 2008-01-07T10:16:12Z by SystemAdmin
      • SystemAdmin
        SystemAdmin
        84 Posts
        ACCEPTED ANSWER

        Re: Support Dynamic Bytecode Enhancement?

        ‏2008-01-07T10:39:45Z  in response to SystemAdmin
        Nice work. There's at least one caveat: the instrumentation code was designed to be run from a single thread. When run as an agent, it may be invoked from several threads, and this can lead to problems, especially when writing coverage files (but maybe other places as well). I now work on making it thread-safe. In the meantime, I guess that to be on the safe side, your Premain method should be all synchronized.

        Yarden Nir-Buchbinder,
        IBM Haifa Research Lab.
        • SystemAdmin
          SystemAdmin
          84 Posts
          ACCEPTED ANSWER

          Re: Support Dynamic Bytecode Enhancement?

          ‏2008-01-07T11:21:49Z  in response to SystemAdmin

          I only register one class loader and, I'd guess, only load one instance of the DynamicInstrumentor (could make it a singleton).

          It's not the premain I'm worried about. It only executes once. The purpose of the premain() is to register an instance of a ClassFileTransformer. All the work of transformation is done in the transform(...) method of that class.

          So really, we just need to synchronized on the transform(...) method, assuming that there is a single instance of it.

          Probably the best thing to do is to make sure there is a single instance of DynamicInstrumentor registered and then synchronize the transform(...) method.

          Of course, I did none of the above!-) Instead, for each call to transform(...) I create a new instance of ClassStreamInstrumentor and nothing is synchronized!-) It works but I'm using it on trivial cases.

          I'll make that change...but I gotta get some sleep (at least an hour or two), I'm flying to London in a few hours from the states and I've got to work on a chapter for a book and prepare to teach a TDD class...

          Have a great week. I'll get around to making this stuff single-threaded to make sure it instruments correctly.













          • SystemAdmin
            SystemAdmin
            84 Posts
            ACCEPTED ANSWER

            Re: Support Dynamic Bytecode Enhancement?

            ‏2008-01-07T11:27:57Z  in response to SystemAdmin
            (Imagine this post has line breaks where appropriate, for some reason this doesn't work)

            Yes, that's what I meant - synchronize the transformation code.

            Have a good time in London!

            Yarden Nir-Buchbinder,
            IBM Haifa Research Lab
            • SystemAdmin
              SystemAdmin
              84 Posts
              ACCEPTED ANSWER

              Re: Support Dynamic Bytecode Enhancement?

              ‏2008-01-08T01:03:39Z  in response to SystemAdmin
              Yep, the line breaks are messed up. I've been using HTML tags to mark my paragraphs!-)
              • SystemAdmin
                SystemAdmin
                84 Posts
                ACCEPTED ANSWER

                Re: Support Dynamic Bytecode Enhancement?

                ‏2008-01-08T06:21:23Z  in response to SystemAdmin
                Like this?

                Oh Yea! Thanks!

                Yarden Nir-Bucbhinder,

                IBM Haifa Research Lab