 | Level: Intermediate Dennis Sosnoski (dms@sosnoski.com), Java and XML consultant, Sosnoski Software Solutions Inc.
04 Oct 2005 JiBX 1.0 uses classworking techniques to enhance the bytecode for compiled classes and directly generate new classes. Bytecode generation has some major advantages over working at the source code level, but it can sometimes get in the way of building and debugging your application. Even aside from issues of convenience, some developers just don't trust anything but "The Source." For JiBX 2.0, lead developer Dennis Sosnoski wants to support both source and bytecode generation techniques. In this article, he discusses some of the differences between source code and bytecode generation techniques and gives his take on how to reconcile the two.
Classworking techniques allow programs to directly manipulate the binary
class representations generated by a compiler from Java™ source code. My JiBX
XML data binding framework is an example of such a program, using classworking
to enhance Java class files with added methods implementing conversions to and
from XML. Working directly with binary classes offers a number of advantages,
including the ability to modify classes for which the source code is not
available. For most purposes, this binary approach works very well.
But sometimes the lack of source code can be a disadvantage. One example of
when source code comes in useful is in debugging. Java debuggers are designed to
work at the source code level, and without Java source code that matches the
bytecode instructions, they're effectively useless. This issue of missing source
code can be a problem in tracking down errors that occur when using frameworks
based on classworking. In the case of JiBX, no debugging information is removed
from the class files -- so you can still debug your original code as usual --
but you're not able to debug through the added methods used for the actual
conversions to and from XML. Even beyond the practical issue of debugging, many
developers are reluctant to trust a framework that manipulates their program at
the bytecode level with no easy way for them to verify the results.
One of the goals I've set for the JiBX 2.0 development is to support source
code enhancement as an alternative to bytecode enhancement. In this article, I
cover both the difficulties of handling these two forms of code in parallel and
the techniques I use to make this work. Along the way, I also discuss
some details of bytecode operation I haven't dealt with in prior articles,
particularly in the areas of method calls and flow of control.
The source alternative
In normal usage, Java source code is translated into bytecode instruction
sequences by a compiler. By contrast, most classworking libraries bypass source
code completely and only work at the bytecode level. The only exception is the
Javassist library, which allows a form of Java source code to be used for
inserting bytecode into methods or constructing new methods (see Resources for links to my earlier Java programming
dynamics articles on Javassist).
One possibility for dual source/bytecode support in JiBX 2.0 might be to
build on Javassist's source code handling by always generating source code and then
using Javassist to convert it to bytecode when class files are being
enhanced directly. But Javassist's source code support is limited and includes
some idiosyncrasies that differ from standard Java source code (including the
way method variables are referenced). Javassist is also slower than some of the
other bytecode libraries (ASM, in particular, as discussed in "Classworking toolkit: ASM Classworking"). I'm expecting
bytecode enhancement to remain the main focus of JiBX 2.0, and in some
circumstances (such as using JiBX in combination with an IDE's automatic
compilation), the bytecode enhancements may need to be done repeatedly, so I see
speed as an important concern. Finally, the Javassist GPL license is not
compatible with JiBX's BSD license. For all these reasons, I'm going to take a
different approach.
My plan is to instead implement the code generation using a strategy
pattern, where the same sorts of operations will be translated differently
depending on whether the source code or bytecode strategy is used. The bytecode
generation is basically the same as the JiBX 1.X implementation (though using
the ASM library, rather than BCEL). The source code generation is new, and it has
to be structured in a way that allows for compatibility at the operation level
with the bytecode generation.
Comparing code forms
Java source code normally gets compiled into bytecode, and some tools can
even turn bytecode (at least the form generated by normal compilers) back into
source code. This convertibility between the two forms of code shows that
there's a high degree of compatibility. Even so, there are still substantial
differences between the programming techniques used in source code and the
bytecode equivalents. In this section, I'll illustrate some of these
differences.
Method parameters and variables
Java source code generally treats method parameters as a special form of
local variables, with the parameter declarations included directly in the method
declaration. There's one exception to this principle, in that virtual methods
use a special first parameter that doesn't appear in the method parameter list.
This hidden parameter is the this reference to the
class instance on which the method is being invoked.
Bytecode also treats method parameters similarly to local variables. In terms
of bytecode, each parameter occupies one or more words of the stack frame in use
when the method is executed. Unlike in source code, everything is explicit in
bytecode -- the this parameter for a virtual method
is always at position 0 in the stack frame, followed by the parameters
explicitly defined in the method declaration. Parameters occupy different
numbers of frame slots depending on the size of the parameter value compared to
the standard word size.
Regular local variables in source code are defined within a block, which may
be the entire method body or some nested block. In bytecode, the same principle
applies. Rather than explicit blocks, though, the bytecode definition of a local
variable defines an instruction range over which the variable is active. The
local variables occupy words of the stack frame, just like the method
parameters. To minimize the amount of stack frame space required for a method, the same words of the stack frame may be used for different local variables at
different points in the bytecode, as long as the active ranges for the variables
don't overlap.
Figure 1 gives an illustration of the stack frame assignments for
a simple method, including the local variables. The long values each take up two words of the frame, while the
int and reference values each take a single word.
Figure 1. Stack frame usage
Method calls and stack use
Method calls look really simple in Java source code: You just write the
method name followed by a comma-delimited list of argument values, the latter
surrounded by parentheses. The argument values are positional and must be
supplied in the same order as the corresponding parameters in the method
declaration. If the method returns a result value, you can assign the value from
the call to a local variable, use the value directly, or just ignore it
completely.
The corresponding bytecode is more complex. Before a method can be called, the
argument values must be pushed on the stack, in left-to-right order
corresponding to the parameter declarations. For a virtual method call (as
opposed to static calls), the
reference to the object instance being called must be pushed before any other
argument values. When all argument values are on the stack, the method can be
called, and after the call, the entire list of argument values will be replaced
by the method return value on the stack. To keep the stack state valid, bytecode
must take the differences between virtual and static methods -- and the return
type -- into account.
I'll illustrate the stack usage with an example. Listing 1 is a class
defining a method similar to that shown in Figure 1. Listing 2
gives a commented version of the bytecode from the beginning of the main() method through the call to the power() method from the Listing 1 class. The Listing 2
lines in bold show the actual power() method call
setup and return handling.
Listing 1. Sample source code
public class PowerTest
{
private long power(long value, int power) {
long result;
if (power < 5) {
// just compute value inline for low loop count
result = 1;
for (int i = 0; i < power; i++) {
result *= value;
}
} else {
// split the computation using recursion for speed
result = power(value, power/2);
result = result*result;
if ((power % 2) == 1) {
result *= value;
}
}
return result;
}
public static void main(String[] args) {
PowerTest inst = new PowerTest();
long value = Long.parseLong(args[0]);
int power = Integer.parseInt(args[1]);
System.out.println(value + " to the power " +
power + " is " + inst.power(value, power));
}
}
|
Listing 2. Commented bytecode for method call
// create and initialize class instance (using default constructor)
new PowerTest
dup
invokespecial PowerTest.<init>
// store reference (duplicated before initializer call) to "inst"
astore_1
// load first command line argument value string from array
aload_0
iconst_0
aaload
// convert and store value to "value"
invokestatic Long.parseLong
lstore_2
// load second command line argument value string from array
aload_0
iconst_1
aaload
// convert and store value to "power"
invokestatic Integer.parseInt
istore %4
// call power() and save result value to "result"
aload_1
lload_2
iload %4
invokespecial PowerTest.power
lstore %5
...
|
Despite the added complexity of bytecode stack manipulation, it also offers
some flexibility that is not available in source code. For instance, bytecode
can handle values that need to be used more than once by duplicating them on
the stack. Getting the same effect in source code requires you to define a local
variable to hold the value. Many types of operations can be structured to take
advantage of the stack usage flexibility provided by bytecode, and JiBX 1.X uses
this flexibility quite heavily in generated code.
Flow of execution
Controlling the normal flow of program execution is also somewhat more
complex in bytecode than in source code. The Java platform provides conditional execution
(using if), three different flavors of looping (for, do, and while), and one fan-out construct (switch). At the bytecode level, there are only two different
basic constructs, one corresponding to switch
statements and the other a branch. But the branch has enough variations to more
than make up for the lower number of basic constructs.
To demonstrate the basic branch operation, Listing 3 shows the commented
bytecode for the Listing 1 power() method. This example includes several branches, with the
bytecode for three branches shown in bold. The
first branch is an if_icmpge conditional. This
branch consumes the top two words from the stack, subtracting the first word
from the second one and taking the branch if the result is non-negative. The
second branch is an unconditional goto. This doesn't
affect the stack, but always transfers to the target offset. The third branch is
an if_icmpne conditional. This branch consumes the
top two words from the stack, subtracting the first word from the second one and
taking the branch if the result is non-zero.
Listing 3. Commented bytecode with branches
// check if "power" less than 5
iload_3
iconst_5
if_icmpge 29
// initialize "result" to 1 and "i" to 0
lconst_1
lstore %4
iconst_0
istore %6
// jump to end if "i" greater than or equal to "power"
11: iload %6
iload_3
if_icmpge 59
// multiply "result" value by "value"
lload %4
lload_1
lmul
lstore %4
// increment "i" value and loop back to test
iinc %6 1
goto 11
// make recursive call for half the "power"
29: aload_0
lload_1
iload_3
iconst_2
idiv
invokespecial PowerTest.power
// square the returned "result" value
lstore %4
lload %4
lload %4
lmul
lstore %4
// check for odd "power" value
iload_3
iconst_2
irem
iconst_1
if_icmpne 59
// odd "power", multiple "result" again for final value
lload %4
lload_1
lmul
lstore %4
// return "result" value
59: lload %4
lreturn
|
Listing 3 demonstrates how Java conditional execution (the if statement) and one form of loop (the for statement) are translated into bytecode. The other
loop constructs in Java source code are handled in much the same way as for. The switch is more
complicated, with bytecode sometimes using normal conditional branches and other
times one of a pair of table-based conditional branches.
Generation strategies
The naive approach to supporting a combination of source code and bytecode
generation is to just provide two completely separate code generation
implementations. This would work, but it would obviously involve a lot of
duplicated effort (both initially, and in maintenance). I'd like to avoid this
duplication of effort.
Rather than duplicating all the generation code, I'm instead implementing a
strategy-type approach that will handle both forms. This uses common code to
control the generation process, calling methods of the strategy implementation
to generate the appropriate code for a particular type of operation. I'll
illustrate this technique with a simple example.
Growing code on trees
In last month's column, I described how the JiBX
1.X binding compiler generates bytecode based on a code generation tree
structure constructed from the binding definition. Handling code generation with
a tree-based approach gives the advantage that it allows all the code to be
generated sequentially -- there's never a need to go back and insert bytecode
into a sequence of instructions generated previously. This keeps the actual
generation code relatively simple.
JiBX 2.0 keeps the principle of working from a tree
representation, though the tree representation used is more directly tied
to the binding definition than with JiBX 1.X. To support both source
and bytecode generation, JiBX 2.0 adds a strategy layer between the sequence
of abstract operations generated from the tree and the actual generated code. To
illustrate how this strategy layer works, I'll walk through part of the code
generation for the JiBX 1.X
model shown in Figure 2 (an example from last month's article), which shows how the
same sequence of abstract operations can be used to generate either bytecode or
source code.
Figure 2. Example code generation model
Abstracting operations
Listing 4 gives a list of abstract operations required to implement
unmarshalling from XML to Java objects for the bottom left portion of the Figure
2 diagram. The top portion of the listing is for unmarshalling a Customer instance, while the bottom portion unmarshals a
Name instance. JiBX normally builds this
unmarshalling code into virtual methods added to the respective classes, using
names that start with "JiBX_" for the added methods. In Listing 4, I've shown
part of the abstract operation sequence for the Customer unmarshal method and the full sequence for the
Name method.
Listing 4. The name field unmarshalling logical operations
load or create object from "name" field, save to local
call Name unmarshalling method
store reference from local variable to "name" field
unmarshal "street1" field value from "street" element
unmarshal "city" field value from "city" element
...
// Name unmarshalling method
call unmarshalling context method to push instance on stack
unmarshal "firstName" field value from "first-name" element
unmarshal "lastName" field value from "last-name" element
call unmarshalling context method to pop instance from stack
|
Generating bytecode
Listing 5 shows bytecode generated from the list of abstract operations
shown in Listing 4. The bytecode assumes the unmarshalling methods added to the
classes are virtual methods taking a single parameter, the JiBX unmarshalling
context being used to interpret the XML input document. Because they're virtual
methods, the first value on the stack frame, at offset 0, is a reference to the
actual object instance. The second value on the stack frame is the unmarshalling
context reference, followed by any local variables used in the code.
Listing 5. The name field unmarshalling bytecode
// load or create object from "name" field, save to local
aload_0
getfield name
dup
astore_3
ifnonnull 20
aload_1
invokestatic Name.JiBX_binding1_newinstance_1_0
astore_3
// call Name unmarshalling method
20: aload_3
aload_1
invokevirtual Name.JiBX_binding1_unmarshal_1_0
// store reference from local variable to "name" field
aload_0
aload_3
putfield name
// unmarshal "street1" field value from "street" element
aload_0
aload_1
aconst_null
ldc "street"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield street1
// unmarshal "city" field value from "city" element
aload_0
aload_1
aconst_null
ldc "city"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield city
...
// Name.JiBX_binding1_unmarshal_1_0 method
// call unmarshalling context method to push instance on stack
aload_1
aload_0
invokevirtual org.jibx...UnmarshallingContext.pushObject
// unmarshal "firstName" field value from "first-name" element
aload_0
aload_1
aconst_null
ldc "first-name"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield firstName
// unmarshal "lastName" field value from "last-name" element
aload_0
aload_1
aconst_null
ldc "last-name"
invokevirtual org.jibx...UnmarshallingContext.parseElementText
putfield lastName
// call unmarshalling context method to pop instance from stack
aload_1
invokevirtual org.jibx...UnmarshallingContext.popObject
return
|
Generating source code
Listing 6 shows Java source code generated from the same list of abstract
operations as used for the Listing 5 bytecode. The variable name "ctx" used in
the source code refers to the unmarshalling context parameter for each
method.
Listing 6. The name field unmarshalling source code
// load or create object from "name" field, save to local
Name local1 = name;
if (local1 == null) {
local1 = Name.JiBX_binding1_newinstance_1_0();
}
// call Name unmarshalling method
local1.JiBX_binding1_unmarshal_1_0(ctx);
// store reference from local variable to "name" field
name = local1;
// unmarshal "street1" field value from "street" element
street1 = ctx.parseElementText(null, "street");
// unmarshal "city" field value from "city" element
city = ctx.parseElementText(null, "city");
...
// Name.JiBX_binding1_unmarshal_1_0 method
// call unmarshalling context method to push instance on stack
ctx.pushObject(this);
// unmarshal "firstName" field value from "first-name" element
firstName = ctx.parseElementText(null, "first-name");
// unmarshal "lastName" field value from "last-name" element
lastName = ctx.parseElementText(null, "last-name");
// call unmarshalling context method to pop instance from stack
ctx.popObject();
|
It's easy to see how both bytecode and source code generation can be
implemented from abstract operations in this simple case. Supporting the full
flexibility of JiBX data binding involves somewhat more complexity, but the
principles remain the same.
Checking changes
The JiBX 1.X binding compiler checks each bound class for methods with names
matching the pattern it uses for binding implementation methods. If the
binding compiler finds methods with appropriate names, it assumes these methods
were added by a previous execution of the binding compiler. When constructing a
new method, the binding compiler first checks for a match with an existing
binding method (either present in the class prior to the binding compiler
execution, or added earlier in the binding compiler processing) before actually
adding the new method to the class. If a match is found, the existing method is
used in place of the newly constructed method. If a binding method originally
present in the class is not used by the binding compiler, it is eliminated from
the class. This method-matching approach both minimizes the amount of code added
by JiBX and prevents unnecessary changes to class files when bindings are
recompiled.
As implemented by JiBX 1.X, the method-matching approach has some
limitations. In particular, it cannot match mutually recursive methods, where
each method calls the other. For JiBX 2.0, I'm hoping to get past this
limitation. However, in this article I'm only covering the basics of how bytecode
method comparison works currently and how I plan to implement equivalent
comparisons for source code methods. This doesn't extend to comparing bytecode
with source code -- that problem is much more involved, but fortunately it's one
that's irrelevant to the needs of JiBX 2.0 users, who will need to specify
whether bytecode or source code modification is to be used for a particular
class.
Bytecode method comparisons
Bytecode method matching in JiBX 1.X is based on comparing the method
signatures, along with the actual binary bytecode instruction sequences for the
methods. Because the binding compiler will always generate the same bytecode
instruction sequence for a binding component, this matching process
works as expected, even for methods that call other methods in the same or
other classes, as long as the called methods are checked for duplication before
the calling method. The tree-based approach used for JiBX code generation ends
up constructing methods in this depth-first order anyway, so the duplicate
checks work as long as there are no cycles in the method call graphs
(mutually recursive methods).
Source code method comparisons
Source code method matching requires a little more work than bytecode method
matching. The nearest equivalent to the bytecode approach of comparing method
signatures and instruction sequences is to match the sequence of Java language
tokens comprising the source code for a method. As with bytecode generation, the
binding compiler will always generate the same source code token sequence for a
binding component. If the token sequence for a newly constructed
method matches that of an existing method, the two are identical, and the
existing method can be used in place of the newly constructed method.
Adding and removing methods is also more complex in source code than in
bytecode. In bytecode, the order of methods within a class representation isn't
of any particular significance. Bytecode methods can be added and removed
without interfering with user code, and generated methods are essentially the
same as those compiled from user code. In source code, on the other hand, user
code may have comments and formatting that do not affect the meaning of the
code but are still important to the user. Anything that distorts the user source
code from its original form is going to create problems.
The standard technique used by frameworks that add generated source code to
user code is to define a delimiter to separate the generated code from the
original code. JiBX 2.0 takes a more flexible approach, initially adding new
methods after existing user methods in a Java source file, but then replacing
code directly if the set of generated methods is modified by later binding
compiles. This approach lets the user move or reformat generated code without interfering
with the method-matching process.
Efficiency can be a potential concern when code needs to scan large numbers
of source code files. JiBX is able to duck this issue because it requires compiled versions of bound
classes to be available even when generating source code enhancements. The
binding compiler is then always able to easily check for JiBX-generated source
method names without processing each and every raw source code file. Only the
source files with added JiBX methods need to be scanned so that the token
sequences for each method can be recorded and used in comparisons.
Generics ahead
This month, I looked at issues involved in letting users choose between
source code enhancement and bytecode enhancement for the binding code generated
by my JiBX XML data binding framework. My conclusions so far are that
-- with the right architecture -- this choice is not too difficult to implement
and will definitely provide a major gain for users. I'm hoping to have a working
beta of JiBX 2.0 out to support this feature by early next year.
Next month, I'm going to look at another aspect of classworking involved in
the JiBX 2.0 changes. Java 5 added generics support to the language, giving
developers a way to both check collection type-safety at compile time and hide
the runtime casting involved with using a collection (though at the cost of a
sometimes very messy syntax for typing). I'm not completely thrilled by the benefits
of generics in writing clean code, but I am very interested in using the added
type information that goes into compiled class files. For my next article, I'll dig into this aspect of generics and
show how you can use reflection to access generic information at run time.
Resources Learn
Get products and technologies
- Javassist: If classworking without learning bytecode sounds like a great combination, Javassist may be just what you're looking for. Javassist lets you work with source code, which it compiles into Java bytecode.
- JiBX XML data binding framework: A fast and flexible XML data binding to your Java classes.
Discuss
About the author | 
| | Dennis Sosnoski is the founder and lead consultant of Seattle-area Java technology
consulting company Sosnoski Software Solutions, Inc., specialists in
XML and Web services training and consulting. His
professional software development experience spans over 30 years, with the last
several years focused on server-side XML and Java technologies. Dennis is a
frequent speaker at conferences nationwide, and a member of the JSR-222 (JAXB
2.0) and JSR-224 (JAX-RPC 2.0) expert groups. He's also the lead developer of the
open source JiBX XML Data Binding framework
built around Java classworking technology. |
Rate this page
| |
