Contents


Custom AST transformations with Project Lombok

How and when to extend Lombok for custom code generation

Comments

Even for die-hard Java™ developers, syntax verbosity can be a downside of writing applications in the Java language. While it's sometimes possible to get around verbosity by using a newer language like Groovy, in some cases using Java code is preferable, perhaps even required. Then you might want to try out Project Lombok, which is an open source, code-generation library for the Java platform.

Lombok does a handy job of reducing line-by-line boilerplate code in Java applications, so there's plenty of Java syntax you'll never write again. But what makes Lombok so sweet isn't just syntactic sugar: it's a unique approach to code generation, and all the Java development possibilities that opens up.

In this article, I introduce Project Lombok and explain why I think it's a great, although not perfect, addition to the Java developer's toolbox. I'll give you an overview of Lombok, including how it works, what it's best used for, and a quick list of its current pros and cons. I'll then focus on one of the most powerful, but also complex, use cases for Lombok: extending it for a custom code base. This could be your own code or an existing Java pattern that isn't yet part of Lombok's library. Either way, the remainder of the article will focus on tips and tricks for extending Lombok, including a few guidelines to determine whether tackling the Lombok APIs is time well spent, or if you'd be better off writing the boilerplate for your particular project after all.

The included sample code (see Download) extends Lombok to generate JavaBeans boilerplate code. It's freely usable and licensed under Apache 2.0.

What makes Lombok different

Perhaps the biggest reason to use Lombok over other code generation tools is that Lombok doesn't just generate Java sources or bytecode: it transforms the Abstract Syntax Tree (AST), by modifying its structure at compile-time. The AST is a tree representation of the parsed source code, created by the compiler, similar to the DOM tree model of an XML file. By modifying (or transforming) the AST, Lombok keeps your source code trim and free of bloat, unlike plain-text code-generation. Lombok's generated code is also visible to classes within the same compilation unit, unlike direct bytecode manipulation with libraries like CGLib or ASM.

Lombok supports more than one mechanism for triggering code generation, including the very popular Java annotations. Using Java annotations allows developers to modify annotated classes, which regular Java annotation processing prohibits.

For an example of what Lombok can do, consider the class in Listing 1:

Listing 1. A simple Java class
public class Person {
  private String firstName;
  private String lastName;
  private int age;
}

It isn't hard to add implementations of equals, hashCode, and toString to your code, it's just tedious and error-prone. You could use a modern Java IDE like Eclipse to automatically generate most of the boilerplate code, but that would be only a partial solution. It would save time and effort, but at the expense of code readability and comprehension, because boilerplate code usually adds noise to application source.

Lombok has a clever solution to the boilerplate code problem, however. Taking Listing 1 as an example, we can easily generate the needed methods by adding the annotation @lombok.Data to the Person.java class. Figure 1 shows Lombok's code generation inside Eclipse. In the Outline view, you can see that the generated methods show up in the compile class, while the source file remains free of boilerplate.

Figure 1. Lombok in action
A screenshot showing that Lombok has added boilerplate code to the Person compile class, but not to the  Person.java source file.
A screenshot showing that Lombok has added boilerplate code to the Person compile class, but not to the Person.java source file.

Lombok's nuts and bolts

Lombok supports the popular Java compilers javac and Eclipse Compiler for Java (ECJ). Even though these two compilers produce similar output, their implementation is completely different. As a result, Lombok ships with two sets of annotation handlers (the code that hooks into Lombok and contains the code generation logic): one for each compiler. Fortunately, this is transparent, so as users we only have to deal with a single set of Java annotations.

Lombok also provides tight integration with Eclipse: Saving a Java file automatically triggers Lombok's code generation (with no perceivable delay) and updates Eclipse's Outline view to show the generated members, as you saw in Figure 1.

For developers who like to look under the hood, Lombok's delombok tool, accessed via the Maven or Ant command line, will be your headlamp. Delombok takes code already transformed by Lombok and generates plain Java source files from it. Code that has been "delombok-ed" will contain all the transformations made previously by Lombok, in plain text. For example, if you applied delombok to the code in Figure 1, you would be able to see how equals, hashCode, and toString were actually implemented.

Nobody's perfect: Downsides of using Lombok

Before you jump on the Lombok wagon and start adding it to your projects, you should know that it does have a few limitations. Two in particular are important:

  • Lombok's strength can be a weakness. The major argument against Lombok is that it performs "too much magic." First, by removing some Java code verbosity, Lombok has changed the thing that many Java programmers love about the language: What you see is what you get. With Lombok, a .java file no longer shows what's in a .class file.

    Second, certain Lombok transformations do change Java syntax as we know it, in a fundamental way. The @SneakyThrows transformation is a good example. It allows you to throw checked exceptions without declaring them in a method definition, as if they were unchecked exceptions:
    Listing 2. @SneakyThrows — pretty sneaky
      // normally, we would need to declare that this method throws Exception
      @SneakyThrows
      public void doSomething() { 
        throw new Exception();
      }
  • Lombok's annotations naming convention doesn't communicate intention. In Lombok, annotations are no longer just metadata: they actually act as commands driving code generation. I believe the annotation @GenerateGetter would communicate its intention better than the current annotation of @Getter.

In addition to these issues with Lombok proper, there also are some issues with its Eclipse integration. For the most part, these are the result of Eclipse not knowing about Lombok's code generation:

  • Eclipse throws NullPointerExceptions from time to time while generating code with Lombok. The source of the problem is still unknown. Closing and reopening Eclipse usually fixes this issue.
  • Refactoring in Eclipse gets trickier with Lombok. For example, to rename a field that has Lombok-generated getters and setters using Eclipse, you need to press Alt-Shift-R twice in order to use the Rename Field dialog instead of performing an in-place field rename. In the Preview step, you have to uncheck the getXXX and setXXX from the type you're refactoring.
  • Because there is no Java source for Lombok-generated code, debugging gets a little bit confusing. For example, if you try to step into the code of a Lombok-generated getter getName, the Eclipse debugger will jump to the @Getter annotation of the field name. Other than that, Eclipse's debugger works just as usual when Lombok is present.

On the whole, these problems can be worked around, and with time most of them will probably be resolved by the Lombok and Eclipse dev teams. Still, it's good to know what you're getting into. This is true anytime you add a new tool to your toolbox.

Extending Lombok

Lombok generates most common Java boilerplate code, including getters, setters, equals, and hashCode, just to name a few. That's useful, but at times you may also want to generate your own boilerplate code. For instance, Lombok doesn't yet support some common coding patterns, like JavaBeans. In some cases, you might also need to generate code that is specific to your project or domain.

The best use case I have found for extending Lombok is prototyping and experimenting with new code patterns at the early stages of a project. As these code patterns mature over time, Lombok makes it very easy to change or enhance their implementation: just modify your annotation handler (the piece of code that hooks into Lombok to generate code) and compile. All the code base will be automatically updated (unless there are changes in public contracts in the generated code, resulting in compilation errors). Once these code patterns have settled down, you have the option to delombok your code. From that point on, you can work with regular Java source.

To extend Lombok, you'll need to identify or create the annotation(s) that will trigger Lombok's code generation. Next, you'll need to write annotation handlers for each of the annotations you've identified. An annotation handler is a class that implements a couple of Lombok interfaces and the AST transformation logic —aka code generation.

The following sections contain recommendations, from project setup to testing, that you may find useful when creating your own AST transformations. I've also included a code sample that demonstrates a functional Lombok extension for JavaBeans support. More about that next.

Lombok generates JavaBeans code

As I previously mentioned, Lombok currently supports common code patterns, but not all of them are covered, including JavaBeans. To demonstrate a Lombok extension, I wrote up a quick sample project that generates JavaBeans plumbing code. Besides showing how to extend Lombok with custom annotation handlers for both javac and ECJ, this project also packs some useful utilities (such as a field and method builder for both compilers) that make the process a lot cleaner and simpler.

I used Eclipse 3.6 (Helios) and a snapshot of Lombok's git repository for version 0.10-BETA2. The code consists of annotation handlers that generate JavaBean "bound" setters. The attached zip file (see the Download section) contains the following:

  • An Ant build file
  • The annotations @GenerateBoundSetter and @GenerateJavaBean
  • Annotations handlers (for both javac and ECJ) that generate the "bound" setter
  • Some JavaBeans plumbing (e.g., generation of a PropertyChangeSupport field)

The attached code is fully functional and is licensed under the Apache 2.0 license. You can obtain an updated version of the code from GitHub (see Related topics). For inspiration, here's a quick look at what the code can do.

If I write the code in Listing 3, Lombok will generate something like the code in Listing 4, using my annotation handlers:

Listing 3. Lombok! Generate JavaBean!
@GenerateJavaBean
public class Person {
  @GenerateBoundSetter private String firstName; 
}
Listing 4. Example of generated JavaBean support code
public class Person {

  public static final String PROP_FIRST_NAME = "firstName";
  
  private String firstName;
 
  private PropertyChangeSupport propertySupport = new PropertyChangeSupport(this);

  public void addPropertyChangeListener(PropertyChangeListener listener) {
    propertySupport.addPropertyChangeListener(listener);
  }

  public void removePropertyChangeListener(PropertyChangeListener listener) {
    propertySupport.removePropertyChangeListener(listener);
  }
  
  public void setFirstName(String value) {
    String oldValue = firstName;
    firstName = value;
    propertySupport.firePropertyChange(PROP_FIRST_NAME, oldValue, firstName);
  }
}

See the readme.txt file included in the sample code for instructions on how to generate an Eclipse project from the sample code's build file.

Getting started: Javac or ECJ?

In my opinion, any Lombok extension needs to support both javac and ECJ, at least for now. Javac is the default compiler used by build tools like Ant and Maven. At the time of this writing, however, Eclipse provides the smoothest code-editing experience when used with Lombok. Supporting both compilers is vital for developer productivity.

Javac and ECJ work with a similar AST structure. Unfortunately they are implemented very differently, forcing you to write two annotation handlers per annotation, one for javac and one for ECJ. The good news is that the Lombok team is already working on a unified AST API, which will eventually allow us to develop a single annotation handler per annotation that works with both compilers (see Related topics).

Studying Lombok's source code

The next thing you'll need to do is see what you're getting into, and for that, there's nothing better than source code.

Lombok uses non-public APIs in both javac and ECJ to accomplish its clever code-generation technique. Because your code will be plugging into Lombok, it is very likely that it will need to use similar, if not the same APIs.

The main problem with non-public APIs is lack of documentation and stability. Fortunately, according to the Lombok team, they haven't had any portability issues with new versions of Eclipse (we will see what happens when Java 7 is released). For the moment, the lack of documentation is the biggest issue you will have to deal with. In addition, even with good documentation, learning the APIs of two different compilers is hard work, and time-consuming. What we need is a "quick and practical guide" to javac and ECJ — something that is out of this article's scope.

The good news is that the Lombok team has done a fairly good job of documenting how they use javac and ECJ to generate AST nodes. I strongly recommend reading their code. They have covered what are the most common use cases: things like variable declarations, method implementations, and so on. Reading Lombok's source code is the fastest way to learn javac's and ECJ's APIs. Listing 5 shows an example of Lombok's own source code:

Listing 5. Generating local variables with Javac
  /* final int PRIME = 31; */ {
    if (!fields.isEmpty() || callSuper) {
      statements.append(maker.VarDef(maker.Modifiers(Flags.FINAL),
          primeName, maker.TypeIdent(Javac.getCTCint(TypeTags.class, "INT")), 
          maker.Literal(31)));
    }
  }

As you can see, the Lombok team has documented what blocks of code generate what. Next time you need to generate the declaration of a local variable, you can go back to this source and use it as reference.

Don't limit yourself to reading just Lombok's .java files. Lombok's developers have also provided good pointers for setting up and building your project and for testing your annotation handlers. I cover these topics in more detail in the following sections.

Dependency management

Once you try automatic dependency management in your projects, it is very hard to go back and do it manually. The Java world has more than one build tool that provides dependency management, including Ivy and Maven (see Related topics). When creating Lombok extensions, the choice narrows down to one, however, and it's Ivy.

One reason for choosing Ivy is that all of the necessary dependencies, like javac, are in Maven's central repository — which pushes Maven out of the picture. The other reason is that Ivy supports managing dependencies that are not in a Maven repository. It is easy to specify a link where a dependency can be downloaded. This configuration requires a custom ivysettings.xml configuration file, which is really not a big deal.

Ivy sits on top of Ant, providing dependency management to your build. The Lombok team use an improved version of Ivy that they have developed themselves, ivyplusplus (see Related topics). This Ivy extension provides some useful Ant targets, such as creating Eclipse and IntelliJ project files from a list of dependencies.

Adding dependencies

To set up your Lombok extension project you will need the following files:

  • build.xml file: An Ant build file that:
    • Downloads ivyplusplus (from a specified location) the first time a build is invoked.
    • Specifies where the Ivy configuration file is.
    • Compiles, tests, and packages your code.
  • buildScripts/ivy.xml file: Specifies your project's dependencies.
  • buildScripts/ivysettings.xml file: Specifies the repositories (Maven or just URLs) from which to get dependencies.
  • buildScripts/ivy-repo folder: Contains an XML file per each dependency specified in ivy.xml. These XML files describe a dependency artifact (for example, a location where it can be downloaded from, home page, etc.)

You don't need to reinvent the wheel. To save time and effort, take a look at the build files from Lombok or from this article's attached source and copy-and-paste the pieces that you need.

Annotation naming

As I mentioned before, Lombok's annotations could do a better job of communicating that they are more than just metadata. They should indicate that they are responsible for triggering some sort of code generation. I therefore strongly suggest that you prepend all your Lombok-related annotations with "Generate." In this article's source code, I have named the annotations that trigger JavaBeans-related source code @GenerateBoundSetter and @GenerateJavaBean. This naming convention at least gives developers unfamiliar with your code base a clue that somewhere in your build environment is a process for generating code.

Documenting AST transformations

Documentation is critical when extending Lombok. Documenting your annotation handlers will benefit maintainers of the AST transformations, while documenting your annotations will benefit their users.

Documenting annotation handlers

Code using javac or ECJ APIs is not trivial to read and understand. It is complex and long, even when it generates the simplest Java code. Documenting your annotation handlers will make their maintenance a lot easier for you and your team. In terms of documentation, I have found that is useful to have the following:

  • A class-level Javadoc comment explaining, at a high level, what code the annotation handler generates. I think the easiest way to explain what code gets generated is to include sample code in that comment, as in Listing 6:
    Listing 6. Class-level Javadoc of an annotation handler
    /**
     * Instructs lombok to generate the necessary code to make an annotated Java 
     * class a JavaBean.
     * <p>
     * For example, given this class:
     * 
     * <pre>
     * @GenerateJavaBean
     * public class Person {
     * 
     * }
     * </pre>
     * our lombok annotation handler (for both javac and eclipse) will generate 
     * the AST nodes that correspond to this code:
     * 
     * <pre>
     * public class Person {
     * 
     *   private PropertyChangeSupport propertySupport 
     *       = new PropertyChangeSupport(this);
     *
     *   public void addPropertyChangeListener(PropertyChangeListener l) {
     *     propertySupport.addPropertyChangeListener(l);
     *   }
     *
     *   public void removePropertyChangeListener(PropertyChangeListener l) {
     *     propertySupport.removePropertyChangeListener(l);
     *   }
     * }
     * </pre>
     * </p>
     *  
     * @author Alex Ruiz
     */
  • Regular, non-javadoc comments through the code base explaining what a block of code generates, as in Listing 7:
    Listing 7. Documenting what a block of code generates
        // public void setFirstName(String value) {
        //   final String oldValue = firstName;
        //   firstName = value;
        //   propertySupport.firePropertyChange(PROP_FIRST_NAME, oldValue, 
        //       firstName);
        // }
        JCVariableDecl fieldDecl = (JCVariableDecl) fieldNode.get();
        long mods = toJavacModifier(accessLevel) | (fieldDecl.mods.flags & STATIC);
        TreeMaker treeMaker = fieldNode.getTreeMaker();
        List<JCAnnotation> nonNulls = findAnnotations(fieldNode, NON_NULL_PATTERN);
        return newMethod().withModifiers(mods)
                          .withName(setterName)
                          .withReturnType(treeMaker.Type(voidType()))
                          .withParameters(parameters(nonNulls, fieldNode))
                          .withBody(body(propertyNameFieldName, fieldNode))
                          .buildWith(fieldNode);

Documenting annotations

Adding a class-level Javadoc comment similar to the one we used for the annotation handler (in Listing 6) helps users of your annotations know and understand what will happen once they consume those annotations.

Consistency between compilers

This tip is useful only if you decide to support both javac and ECJ. When you have two sets of annotation handlers, any bug fix, change, or addition should be applied to both sets (or branches). The more similar the branches are, the faster and safer it will be to make such changes. That similarity must happen at both the package level and the file level.

Package level consistency: As much as possible, each branch (javac and ECJ) should have the same number of classes, using the same name, as shown in Figure 2:

Figure 2. Package similarities between javac and ECJ branches
A screenshot showing package similarities between javac and ECJ packages.

File level consistency: Because both branches will have more or less a similar number of classes with similar names, content in each pair of files with the same name must be as similar as possible: fields, method count, method names, etc., should all be nearly the same. Listing 8 shows the method generatePropertySupportField for both javac and ECJ. Note that even though the AST APIs are different, the implementation of these methods is very similar.

Listing 8. Comparing javac and ECJ annotation handlers
// javac
  private void generatePropertyChangeSupportField(JavacNode typeNode) {
    if (fieldAlreadyExists(PROPERTY_SUPPORT_FIELD_NAME, typeNode)) return;
    JCExpression exprForThis = chainDots(typeNode.getTreeMaker(), typeNode, "this");
    JCVariableDecl fieldDecl = newField().ofType(PropertyChangeSupport.class)
                                         .withName(PROPERTY_SUPPORT_FIELD_NAME)
                                         .withModifiers(PRIVATE | FINAL)
                                         .withArgs(exprForThis)
                                         .buildWith(typeNode);
    injectField(typeNode, fieldDecl);
  }

// ECJ
  private void generatePropertyChangeSupportField(EclipseNode typeNode) {
    if (fieldAlreadyExists(PROPERTY_SUPPORT_FIELD_NAME, typeNode)) return;
    Expression exprForThis = referenceForThis(typeNode.get());
    FieldDeclaration fieldDecl = newField().ofType(PropertyChangeSupport.class)
                                           .withName(PROPERTY_SUPPORT_FIELD_NAME)
                                           .withModifiers(PRIVATE | FINAL)
                                           .withArgs(exprForThis)
                                           .buildWith(typeNode);
    injectField(typeNode, fieldDecl);
  }

Testing AST transformations

Testing custom AST transformations is a lot easier than you might expect, thanks to the testing infrastructure that Lombok has in place. To demonstrate how easy it is to test AST transformations, let's start with the JUnit test case in Listing 9:

Listing 9. Unit test for all ECJ annotation handlers
import static lombok.DirectoryRunner.Compiler.ECJ;

import java.io.File;

import lombok.*;
import lombok.DirectoryRunner.Compiler;
import lombok.DirectoryRunner.TestParams;

import org.junit.runner.RunWith;

/**
 * @author Alex Ruiz
 */
@RunWith(DirectoryRunner.class)
public class TestWithEcj implements TestParams {

  @Override public Compiler getCompiler() {
    return ECJ;
  }

  @Override public boolean printErrors() {
    return true;
  }

  @Override public File getBeforeDirectory() {
    return new File("test/transform/resource/before");
  }

  @Override public File getAfterDirectory() {
    return new File("test/transform/resource/after-ecj");
  }

  @Override public File getMessagesDirectory() {
    return new File("test/transform/resource/messages-ecj");
  }
}

This test works, more or less, as follows:

  1. The test compiles all the Java files in the folder specified by getBeforeDirectory, using the compiler specified by getCompiler) and Lombok.
  2. After compilation is done, the test creates a text representation of the compiled classes using delombok.
  3. The test reads the files from the folder specified in getAfterDirectory. These files contain the expected content of the compiled classes. The test compares the content of such files against the source obtained [in Step 2]. The files to compare must have the same name.
  4. The test reads the files from the folder specified in getMessagesDirectory. These files contain the expected compiler messages (warnings and errors). The test compares the content of such files against the actual messages shown during compilation, if any. It is not necessary to have a message file if for a compilation of a Java file, there are no expected messages. The matching is done by name. For example, if there are expected compiler messages when compiling CompleteJavaBean.java, the file containing such messages should be named CompleteJavaBean.java.messages.
  5. If all the expected outputs match the actual ones, the test passes; otherwise, it fails.

As you can see, this a very different but effective way to test annotation handlers:

  • There is one JUnit test per compiler (javac and ECJ) instead of one JUnit test per annotation handler.
  • Instead of having a test method per use case, we have a text file containing the expected generated code and a optional text file containing the expected compiler messages.
  • The tests do not care how javac and ECJ APIs are being used. The tests verify that the generated code is correct.

Verifying generated code

The test I've described does a good job at verifying that your annotation handlers generate the code you expect. You still need to test that the generated code actually does what you expect it to do, however. To verify the correctness of the generated code's behavior, you'll need to write Java classes that use your AST transformations, then write tests that check the behavior of the generated code. Basically, you'll be testing the code as if it was written by you.

The easiest way to compile and run those tests is by using Ant, which means compiling with javac. Because you already tested and know that the code generated while using ECJ is correct, I don't think it is necessary to run these tests inside of Eclipse (which can complicate setup enormously).

I've included tests for both javac and ECJ annotation handlers in this article's sample code (see Download).

In conclusion

Project Lombok is a powerful tool that successfully reduces the verbosity of Java code. It accomplishes this goal with some clever and unusual uses of Java annotations and compiler APIs. Like any tool, it is not perfect. The gains (shorter and cleaner code) come at a cost: Java code loses its WYSIWYG feel, and developers lose some of the IDE functionality that we love. Before adding Lombok to your toolbox, definitely consider its pros and cons, and determine for yourself if the gains justify the loses.

If you decide to use Lombok, there's a chance that you might want to extend it to generate your own boilerplate code. Currently, extending Lombok is not an easy task, nor is it for everyone, but it is doable. This article has presented some guidelines regarding when to extend Lombok and described how to do it. It is up to you to determine if the time and effort you will spend extending Lombok is cheaper than just writing boilerplate code by hand.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java development, Open source
ArticleID=629124
ArticleTitle=Custom AST transformations with Project Lombok
publish-date=03012011