Classworking toolkit: Combining source and bytecode generation

Source code generation and bytecode generation each have advantages -- why not offer a choice?

JiBX 1.0 uses classworking techniques to enhance the bytecode for compiled classes and directly generate new classes. Bytecode generation has some major advantages over working at the source code level, but it can sometimes get in the way of building and debugging your application. Even aside from issues of convenience, some developers just don't trust anything but "The Source." For JiBX 2.0, lead developer Dennis Sosnoski wants to support both source and bytecode generation techniques. In this article, he discusses some of the differences between source code and bytecode generation techniques and gives his take on how to reconcile the two.

Dennis Sosnoski (dms@sosnoski.com), Java and XML consultant, Sosnoski Software Solutions Inc.

Dennis SosnoskiDennis Sosnoski is the founder and lead consultant of Seattle-area Java technology consulting company Sosnoski Software Solutions, Inc., specialists in XML and Web services training and consulting. His professional software development experience spans over 30 years, with the last several years focused on server-side XML and Java technologies. Dennis is a frequent speaker at conferences nationwide, and a member of the JSR-222 (JAXB 2.0) and JSR-224 (JAX-RPC 2.0) expert groups. He's also the lead developer of the open source JiBX XML Data Binding framework built around Java classworking technology.



04 October 2005

Classworking techniques allow programs to directly manipulate the binary class representations generated by a compiler from Java™ source code. My JiBX XML data binding framework is an example of such a program, using classworking to enhance Java class files with added methods implementing conversions to and from XML. Working directly with binary classes offers a number of advantages, including the ability to modify classes for which the source code is not available. For most purposes, this binary approach works very well.

But sometimes the lack of source code can be a disadvantage. One example of when source code comes in useful is in debugging. Java debuggers are designed to work at the source code level, and without Java source code that matches the bytecode instructions, they're effectively useless. This issue of missing source code can be a problem in tracking down errors that occur when using frameworks based on classworking. In the case of JiBX, no debugging information is removed from the class files -- so you can still debug your original code as usual -- but you're not able to debug through the added methods used for the actual conversions to and from XML. Even beyond the practical issue of debugging, many developers are reluctant to trust a framework that manipulates their program at the bytecode level with no easy way for them to verify the results.

One of the goals I've set for the JiBX 2.0 development is to support source code enhancement as an alternative to bytecode enhancement. In this article, I cover both the difficulties of handling these two forms of code in parallel and the techniques I use to make this work. Along the way, I also discuss some details of bytecode operation I haven't dealt with in prior articles, particularly in the areas of method calls and flow of control.

The source alternative

In normal usage, Java source code is translated into bytecode instruction sequences by a compiler. By contrast, most classworking libraries bypass source code completely and only work at the bytecode level. The only exception is the Javassist library, which allows a form of Java source code to be used for inserting bytecode into methods or constructing new methods (see Resources for links to my earlier Java programming dynamics articles on Javassist).

One possibility for dual source/bytecode support in JiBX 2.0 might be to build on Javassist's source code handling by always generating source code and then using Javassist to convert it to bytecode when class files are being enhanced directly. But Javassist's source code support is limited and includes some idiosyncrasies that differ from standard Java source code (including the way method variables are referenced). Javassist is also slower than some of the other bytecode libraries (ASM, in particular, as discussed in "Classworking toolkit: ASM Classworking"). I'm expecting bytecode enhancement to remain the main focus of JiBX 2.0, and in some circumstances (such as using JiBX in combination with an IDE's automatic compilation), the bytecode enhancements may need to be done repeatedly, so I see speed as an important concern. Finally, the Javassist GPL license is not compatible with JiBX's BSD license. For all these reasons, I'm going to take a different approach.

My plan is to instead implement the code generation using a strategy pattern, where the same sorts of operations will be translated differently depending on whether the source code or bytecode strategy is used. The bytecode generation is basically the same as the JiBX 1.X implementation (though using the ASM library, rather than BCEL). The source code generation is new, and it has to be structured in a way that allows for compatibility at the operation level with the bytecode generation.

Ask the expert: Dennis Sosnoski on JVM and bytecode issues

For comments or questions about the material covered in this article series, as well as anything else that pertains to Java bytecode, the Java binary class format, or general JVM issues, visit the JVM and Bytecode discussion forum, moderated by Dennis Sosnoski.

Comparing code forms

Java source code normally gets compiled into bytecode, and some tools can even turn bytecode (at least the form generated by normal compilers) back into source code. This convertibility between the two forms of code shows that there's a high degree of compatibility. Even so, there are still substantial differences between the programming techniques used in source code and the bytecode equivalents. In this section, I'll illustrate some of these differences.

Method parameters and variables

Java source code generally treats method parameters as a special form of local variables, with the parameter declarations included directly in the method declaration. There's one exception to this principle, in that virtual methods use a special first parameter that doesn't appear in the method parameter list. This hidden parameter is the this reference to the class instance on which the method is being invoked.

Bytecode also treats method parameters similarly to local variables. In terms of bytecode, each parameter occupies one or more words of the stack frame in use when the method is executed. Unlike in source code, everything is explicit in bytecode -- the this parameter for a virtual method is always at position 0 in the stack frame, followed by the parameters explicitly defined in the method declaration. Parameters occupy different numbers of frame slots depending on the size of the parameter value compared to the standard word size.

Regular local variables in source code are defined within a block, which may be the entire method body or some nested block. In bytecode, the same principle applies. Rather than explicit blocks, though, the bytecode definition of a local variable defines an instruction range over which the variable is active. The local variables occupy words of the stack frame, just like the method parameters. To minimize the amount of stack frame space required for a method, the same words of the stack frame may be used for different local variables at different points in the bytecode, as long as the active ranges for the variables don't overlap.

Figure 1 gives an illustration of the stack frame assignments for a simple method, including the local variables. The long values each take up two words of the frame, while the int and reference values each take a single word.

Figure 1. Stack frame usage
Stack frame usage

Method calls and stack use

Method calls look really simple in Java source code: You just write the method name followed by a comma-delimited list of argument values, the latter surrounded by parentheses. The argument values are positional and must be supplied in the same order as the corresponding parameters in the method declaration. If the method returns a result value, you can assign the value from the call to a local variable, use the value directly, or just ignore it completely.

The corresponding bytecode is more complex. Before a method can be called, the argument values must be pushed on the stack, in left-to-right order corresponding to the parameter declarations. For a virtual method call (as opposed to static calls), the reference to the object instance being called must be pushed before any other argument values. When all argument values are on the stack, the method can be called, and after the call, the entire list of argument values will be replaced by the method return value on the stack. To keep the stack state valid, bytecode must take the differences between virtual and static methods -- and the return type -- into account.

I'll illustrate the stack usage with an example. Listing 1 is a class defining a method similar to that shown in Figure 1. Listing 2 gives a commented version of the bytecode from the beginning of the main() method through the call to the power() method from the Listing 1 class. The Listing 2 lines in bold show the actual power() method call setup and return handling.

Listing 1. Sample source code
public class PowerTest
{
    private long power(long value, int power) {
        long result;
        if (power < 5) {
            
            // just compute value inline for low loop count
            result = 1;
            for (int i = 0; i < power; i++) {
                result *= value;
            }
            
        } else {
            
            // split the computation using recursion for speed
            result = power(value, power/2);
            result = result*result;
            if ((power % 2) == 1) {
                result *= value;
            }
            
        }
        return result;
    }
    
    public static void main(String[] args) {
        PowerTest inst = new PowerTest();
        long value = Long.parseLong(args[0]);
        int power = Integer.parseInt(args[1]);
        System.out.println(value + " to the power " + 
          power + " is " + inst.power(value, power));
    }
}
Listing 2. Commented bytecode for method call
// create and initialize class instance (using default constructor)
new   PowerTest
dup
invokespecial PowerTest.<init>
    
// store reference (duplicated before initializer call) to "inst"
astore_1
    
// load first command line argument value string from array
aload_0
iconst_0
aaload
    
// convert and store value to "value"
invokestatic  Long.parseLong
lstore_2
    
// load second command line argument value string from array
aload_0
iconst_1
aaload
    
// convert and store value to "power"
invokestatic  Integer.parseInt
istore  %4
    
// call power() and save result value to "result"
aload_1lload_2iload   %4invokespecial PowerTest.powerlstore  %5
 ...

Despite the added complexity of bytecode stack manipulation, it also offers some flexibility that is not available in source code. For instance, bytecode can handle values that need to be used more than once by duplicating them on the stack. Getting the same effect in source code requires you to define a local variable to hold the value. Many types of operations can be structured to take advantage of the stack usage flexibility provided by bytecode, and JiBX 1.X uses this flexibility quite heavily in generated code.

Flow of execution

Controlling the normal flow of program execution is also somewhat more complex in bytecode than in source code. The Java platform provides conditional execution (using if), three different flavors of looping (for, do, and while), and one fan-out construct (switch). At the bytecode level, there are only two different basic constructs, one corresponding to switch statements and the other a branch. But the branch has enough variations to more than make up for the lower number of basic constructs.

To demonstrate the basic branch operation, Listing 3 shows the commented bytecode for the Listing 1 power() method. This example includes several branches, with the bytecode for three branches shown in bold. The first branch is an if_icmpge conditional. This branch consumes the top two words from the stack, subtracting the first word from the second one and taking the branch if the result is non-negative. The second branch is an unconditional goto. This doesn't affect the stack, but always transfers to the target offset. The third branch is an if_icmpne conditional. This branch consumes the top two words from the stack, subtracting the first word from the second one and taking the branch if the result is non-zero.

Listing 3. Commented bytecode with branches
// check if "power" less than 5
iload_3
iconst_5
if_icmpge 29
    
// initialize "result" to 1 and "i" to 0
lconst_1
lstore  %4
iconst_0
istore  %6
    
// jump to end if "i" greater than or equal to "power"
11: iload   %6iload_3if_icmpge 59
    
// multiply "result" value by "value"
lload   %4
lload_1
lmul
lstore  %4
    
// increment "i" value and loop back to test
iinc    %6 1
goto  11
    
// make recursive call for half the "power"
29: aload_0
lload_1
iload_3
iconst_2
idiv
invokespecial PowerTest.power
    
// square the returned "result" value
lstore  %4
lload   %4
lload   %4
lmul
lstore  %4
    
// check for odd "power" value
iload_3
iconst_2
irem
iconst_1if_icmpne 59
    
// odd "power", multiple "result" again for final value
lload %4
lload_1
lmul
lstore  %4
    
// return "result" value
59: lload   %4
lreturn

Listing 3 demonstrates how Java conditional execution (the if statement) and one form of loop (the for statement) are translated into bytecode. The other loop constructs in Java source code are handled in much the same way as for. The switch is more complicated, with bytecode sometimes using normal conditional branches and other times one of a pair of table-based conditional branches.


Generation strategies

The naive approach to supporting a combination of source code and bytecode generation is to just provide two completely separate code generation implementations. This would work, but it would obviously involve a lot of duplicated effort (both initially, and in maintenance). I'd like to avoid this duplication of effort.

Rather than duplicating all the generation code, I'm instead implementing a strategy-type approach that will handle both forms. This uses common code to control the generation process, calling methods of the strategy implementation to generate the appropriate code for a particular type of operation. I'll illustrate this technique with a simple example.

Growing code on trees

In last month's column, I described how the JiBX 1.X binding compiler generates bytecode based on a code generation tree structure constructed from the binding definition. Handling code generation with a tree-based approach gives the advantage that it allows all the code to be generated sequentially -- there's never a need to go back and insert bytecode into a sequence of instructions generated previously. This keeps the actual generation code relatively simple.

JiBX 2.0 keeps the principle of working from a tree representation, though the tree representation used is more directly tied to the binding definition than with JiBX 1.X. To support both source and bytecode generation, JiBX 2.0 adds a strategy layer between the sequence of abstract operations generated from the tree and the actual generated code. To illustrate how this strategy layer works, I'll walk through part of the code generation for the JiBX 1.X model shown in Figure 2 (an example from last month's article), which shows how the same sequence of abstract operations can be used to generate either bytecode or source code.

Figure 2. Example code generation model
Example code generation model

Abstracting operations

Listing 4 gives a list of abstract operations required to implement unmarshalling from XML to Java objects for the bottom left portion of the Figure 2 diagram. The top portion of the listing is for unmarshalling a Customer instance, while the bottom portion unmarshals a Name instance. JiBX normally builds this unmarshalling code into virtual methods added to the respective classes, using names that start with "JiBX_" for the added methods. In Listing 4, I've shown part of the abstract operation sequence for the Customer unmarshal method and the full sequence for the Name method.

Listing 4. The name field unmarshalling logical operations
load or create object from "name" field, save to local
call Name unmarshalling method
store reference from local variable to "name" field
unmarshal "street1" field value from "street" element
unmarshal "city" field value from "city" element
...
    
// Name unmarshalling method
call unmarshalling context method to push instance on stack
unmarshal "firstName" field value from "first-name" element
unmarshal "lastName" field value from "last-name" element
call unmarshalling context method to pop instance from stack

Generating bytecode

Listing 5 shows bytecode generated from the list of abstract operations shown in Listing 4. The bytecode assumes the unmarshalling methods added to the classes are virtual methods taking a single parameter, the JiBX unmarshalling context being used to interpret the XML input document. Because they're virtual methods, the first value on the stack frame, at offset 0, is a reference to the actual object instance. The second value on the stack frame is the unmarshalling context reference, followed by any local variables used in the code.

Listing 5. The name field unmarshalling bytecode
// load or create object from "name" field, save to local
aload_0
getfield  name
dup
astore_3
ifnonnull 20
aload_1
invokestatic  Name.JiBX_binding1_newinstance_1_0
astore_3
    
// call Name unmarshalling method
20: aload_3
aload_1
invokevirtual Name.JiBX_binding1_unmarshal_1_0
    
// store reference from local variable to "name" field
aload_0
aload_3
putfield  name
    
// unmarshal "street1" field value from "street" element
aload_0
aload_1
aconst_null
ldc "street"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield  street1
    
// unmarshal "city" field value from "city" element
aload_0
aload_1
aconst_null
ldc "city"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield  city
...

// Name.JiBX_binding1_unmarshal_1_0 method
// call unmarshalling context method to push instance on stack
aload_1
aload_0
invokevirtual org.jibx...UnmarshallingContext.pushObject
    
// unmarshal "firstName" field value from "first-name" element
aload_0
aload_1
aconst_null
ldc "first-name"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield  firstName
    
// unmarshal "lastName" field value from "last-name" element
aload_0
aload_1
aconst_null
ldc "last-name"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield  lastName
    
// call unmarshalling context method to pop instance from stack
aload_1
invokevirtual org.jibx...UnmarshallingContext.popObject
    
return

Generating source code

Listing 6 shows Java source code generated from the same list of abstract operations as used for the Listing 5 bytecode. The variable name "ctx" used in the source code refers to the unmarshalling context parameter for each method.

Listing 6. The name field unmarshalling source code
// load or create object from "name" field, save to local
Name local1 = name;
if (local1 == null) {
    local1 = Name.JiBX_binding1_newinstance_1_0();
}
    
// call Name unmarshalling method
local1.JiBX_binding1_unmarshal_1_0(ctx);
    
// store reference from local variable to "name" field
name = local1;
    
// unmarshal "street1" field value from "street" element
street1 = ctx.parseElementText(null, "street");
    
// unmarshal "city" field value from "city" element
city = ctx.parseElementText(null, "city");
...

// Name.JiBX_binding1_unmarshal_1_0 method
// call unmarshalling context method to push instance on stack
ctx.pushObject(this);
    
// unmarshal "firstName" field value from "first-name" element
firstName = ctx.parseElementText(null, "first-name");
    
// unmarshal "lastName" field value from "last-name" element
lastName = ctx.parseElementText(null, "last-name");
    
// call unmarshalling context method to pop instance from stack
ctx.popObject();

It's easy to see how both bytecode and source code generation can be implemented from abstract operations in this simple case. Supporting the full flexibility of JiBX data binding involves somewhat more complexity, but the principles remain the same.


Checking changes

The JiBX 1.X binding compiler checks each bound class for methods with names matching the pattern it uses for binding implementation methods. If the binding compiler finds methods with appropriate names, it assumes these methods were added by a previous execution of the binding compiler. When constructing a new method, the binding compiler first checks for a match with an existing binding method (either present in the class prior to the binding compiler execution, or added earlier in the binding compiler processing) before actually adding the new method to the class. If a match is found, the existing method is used in place of the newly constructed method. If a binding method originally present in the class is not used by the binding compiler, it is eliminated from the class. This method-matching approach both minimizes the amount of code added by JiBX and prevents unnecessary changes to class files when bindings are recompiled.

As implemented by JiBX 1.X, the method-matching approach has some limitations. In particular, it cannot match mutually recursive methods, where each method calls the other. For JiBX 2.0, I'm hoping to get past this limitation. However, in this article I'm only covering the basics of how bytecode method comparison works currently and how I plan to implement equivalent comparisons for source code methods. This doesn't extend to comparing bytecode with source code -- that problem is much more involved, but fortunately it's one that's irrelevant to the needs of JiBX 2.0 users, who will need to specify whether bytecode or source code modification is to be used for a particular class.

Bytecode method comparisons

Bytecode method matching in JiBX 1.X is based on comparing the method signatures, along with the actual binary bytecode instruction sequences for the methods. Because the binding compiler will always generate the same bytecode instruction sequence for a binding component, this matching process works as expected, even for methods that call other methods in the same or other classes, as long as the called methods are checked for duplication before the calling method. The tree-based approach used for JiBX code generation ends up constructing methods in this depth-first order anyway, so the duplicate checks work as long as there are no cycles in the method call graphs (mutually recursive methods).

Source code method comparisons

Source code method matching requires a little more work than bytecode method matching. The nearest equivalent to the bytecode approach of comparing method signatures and instruction sequences is to match the sequence of Java language tokens comprising the source code for a method. As with bytecode generation, the binding compiler will always generate the same source code token sequence for a binding component. If the token sequence for a newly constructed method matches that of an existing method, the two are identical, and the existing method can be used in place of the newly constructed method.

Adding and removing methods is also more complex in source code than in bytecode. In bytecode, the order of methods within a class representation isn't of any particular significance. Bytecode methods can be added and removed without interfering with user code, and generated methods are essentially the same as those compiled from user code. In source code, on the other hand, user code may have comments and formatting that do not affect the meaning of the code but are still important to the user. Anything that distorts the user source code from its original form is going to create problems.

The standard technique used by frameworks that add generated source code to user code is to define a delimiter to separate the generated code from the original code. JiBX 2.0 takes a more flexible approach, initially adding new methods after existing user methods in a Java source file, but then replacing code directly if the set of generated methods is modified by later binding compiles. This approach lets the user move or reformat generated code without interfering with the method-matching process.

Efficiency can be a potential concern when code needs to scan large numbers of source code files. JiBX is able to duck this issue because it requires compiled versions of bound classes to be available even when generating source code enhancements. The binding compiler is then always able to easily check for JiBX-generated source method names without processing each and every raw source code file. Only the source files with added JiBX methods need to be scanned so that the token sequences for each method can be recorded and used in comparisons.


Generics ahead

This month, I looked at issues involved in letting users choose between source code enhancement and bytecode enhancement for the binding code generated by my JiBX XML data binding framework. My conclusions so far are that -- with the right architecture -- this choice is not too difficult to implement and will definitely provide a major gain for users. I'm hoping to have a working beta of JiBX 2.0 out to support this feature by early next year.

Next month, I'm going to look at another aspect of classworking involved in the JiBX 2.0 changes. Java 5 added generics support to the language, giving developers a way to both check collection type-safety at compile time and hide the runtime casting involved with using a collection (though at the cost of a sometimes very messy syntax for typing). I'm not completely thrilled by the benefits of generics in writing clean code, but I am very interested in using the added type information that goes into compiled class files. For my next article, I'll dig into this aspect of generics and show how you can use reflection to access generic information at run time.

Resources

Learn

Get products and technologies

  • Javassist: If classworking without learning bytecode sounds like a great combination, Javassist may be just what you're looking for. Javassist lets you work with source code, which it compiles into Java bytecode.
  • JiBX XML data binding framework: A fast and flexible XML data binding to your Java classes.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology
ArticleID=94790
ArticleTitle=Classworking toolkit: Combining source and bytecode generation
publish-date=10042005