This article assumes a rudimentary familiarity with Eclipse. We will talk about the eBay Development Environment (EDE), which is based on Eclipse V3.2 and requires JDK V5.0 or later (see Resources for links). Some familiarity with Web development, including HTML, JavaScript, and CSS, is useful, but not required.

eBay's challenge

Software development comes in all shapes and sizes. It is a hobby for many folks. It is a great business for doing consulting. It is also done by organizations, both very small and very large. When you are developing software on your own, you pick the tools you like. You write code the way you like. You keep track of your code and maybe perform some backups periodically. As soon as you start working with others, things become trickier. This equation only gets more complicated when the development is being done by an organization of thousands. Luckily, Eclipse is up to this challenge. It provides a powerful tool for standardizing around. The extensibility afforded by its plug-in architecture makes it easy to tailor its functionality to your organization, regardless of its size. To demonstrate, we will look at how eBay has used Eclipse to meet its needs.

The eBay organization

You have almost certainly heard of eBay. Millions of people use it to buy and sell merchandise each year. The eBay marketplace is a highly diversified e-commerce platform that handles a huge number of transactions each second. To accomplish this, eBay employs thousands of software engineers who crank out millions of lines of code. This presents several challenges, such as managing large-scale development, integrating source-code management, and creating code consistency and quality. Let's take a look at each of these and see how eBay has used Eclipse to address these challenges.

Large-scale development

All of those engineers produce lots of code. This code is not one-off throwaway code. It has to be maintained by the same large organization that developed it. That means the same code written by one developer will often be used and modified by many other engineers. The only approach that will scale is to standardize — standardize the code organization, standardize coding conventions, and standardize the tools. Standardize whatever you can.

One of the first problems facing an organization with a large code base is how to organize the code. When you have thousands of packages and classes, you can have an incredible array of dependencies. Imagine how hard it could be just to set up a project with the right classpath. Now imagine thousands of developers having to do this just to get started. Lots of time could be wasted just trying to get existing code to compile so it can be maintained or built upon.

This problem has been addressed at eBay through automation using an Eclipse plug-in. eBay keeps track of projects and their interdependencies using a proprietary, but very simple, XML configuration mechanism. Each file organizes the code into a project and specifies what other projects it depends on. An example of such a file is shown in Listing 1.

                
<project
    type='JAR' 
    path='v4samplecode' 
    javac.source='1.5' 
    javac.target='1.5'
    hasV4Js='true'
    hasV4Css="true"
    >
    <src>
        <dir path='src'/>
    </src>
    <dependencies>
        <project name='GlobalEnvironment'/>
        <project name='DocumentFramework'/>    
        <project name='DarwinTestUtils'/>
        <project name='DarwinCoreUtils'/>
        <project name='DarwinCoreComponents'/>
        <project name='DarwinCoreApp'/>
        <project name='DarwinCoreAppResources'/>
        <project name='V4DeCodeContent' />
        <project name='DarwinTools' />
        <project name='DarwinCodeGenImpl' />
        <library name='DSF-Foo' />
        <library name='EbayResources'/>
        <project name='EbayUrlServices'/>
        <library name='Darwin'/>
        <library name='DarwinTests'/>
        <library name='Dsf'/>
        <library name='Kernel'/>
        <library name="KernelTestAPI"/>
        <library name='xml'/>
        <library name='commons-jxpath'/>
        <library name='icesoft-sdk'/>
        <library name='junit'/>
        <library name='servlet-api'/>
        <library name='jsp-api'/>
        <library name="jakarta-taglibs-standard"/>
    </dependencies>
</project>

This file allows the eBay plug-in to generate Eclipse projects (.project and .classpath files) and to reference other Eclipse projects in the classpath. Obviously, these files are extremely important, and eBay has created a plug-in for helping edit and visualize them.

The XML files are checked into source control (more on that later), then the eBay plug-in creates all the Eclipse projects and imports them into a developer's workspace. People can click one button, as shown in Figure 2.

Like the tool tip suggests, the build plug-in does a lot.

Clicking OK creates all the Eclipse projects and builds them. An example of a relatively small workspace is shown below.

You can only imagine how painful it would be if engineers had to set up all these projects themselves, just so they could create a project. Such is the complexity in working with a large code base. The eBay solution is no silver bullet. It is nontrivial to create a large number of projects with interdependencies and then build all of those projects into a deliverable. The eBay plug-in just automates all of this. However, if you have worked with Eclipse on large projects, you know that even having everything well organized does not solve all problems. Just having a huge number of files and classes can become a problem. This is another challenge eBay has addressed with the help of an Eclipse plug-in.

Back to top

Megajars

As you saw above, it is easy to wind up with a project that uses a huge number of classes from a large code base. Have you ever tried to run Eclipse with a few hundred-thousand source files? It can put a huge strain on your system, requiring lots of memory (ever get an out-of-memory exception from Eclipse?) and causing everything to run extremely slowly. In such a system, it is much better to link against the compiled code instead of the actual source code. This is done at eBay using what eBay calls Megajars.

The idea is really quite simple. Imagine your project depends a dozen or so other projects, which have their own dependency trees. You are not going to change any of those projects. You just depend on them to build your project. So what you want to do is compile all of that source code into a JAR. You don't want to lose the benefits of being able to access the JavaDoc on that code from within Eclipse, but Eclipse makes it easy to associate source code to a JAR for that very purpose.

This is how Megajars work. You provide a set of code to build against, and the eBay build system provides you with the corresponding Megajar. The build plug-in then builds your workspace against the Megajar instead of the source code. Of course, this is all done by an Eclipse plug-in. There is a Megajars view, as shown in Figure 5.

Clicking the Megajar icon initiates the Megajar process. You get an info box telling you exactly what is going to happen.

This is a one-time step that can take a while, but is worth it. It removes the strain on your computer and makes building and deploying your system much faster. It saves time and saves developers from the frustrations of having a slow system. It short, it really helps with a massive code base from a code-organization standpoint. The eBay team has created some other tools to help developers work with their large code base. These tools use visualization to make it easier for eBay developers to find their way around large code bases.

Back to top

Managing scale: Visualization and search tools

In any software application, there are certain files/classes that have extra significance. This is true even when the code involved has a huge numbers of classes. The big difference is that these classes become harder to find. Fortunately, this is the kind of problem tools can really help with.

In Part 1, we talk about eBay's V4 architecture. This is a front-end framework developed by eBay to address the challenges of developing large-scale Web applications. One of the important constructs in V4 is (not too surprisingly) called an application spec. A single project could create numerous Web pages, each with their own content, CSS, JavaScript, etc. All of these various pages and resources can lead to a large number of classes, but there may be a single application spec. Luckily, eBay has built a tool for finding and managing them: The Application Explorer is shown below.

The Application Explorer can identify the application specs in a given project quickly and show visualizations of the meta-data for each application spec. It can be accessed by simply right-clicking on a project in the normal Eclipse explorer.

The application spec is a pretty high-level object in the V4 architecture. An application could have multiple pages. Each page could be built using low-level APIs, adding divs, spans, forms, etc. However, V4 is a component architecture, and most pages are actually built from such components.

The reuse of components boosts eBay developer productivity. There are several keys to component reuse. Obviously, you must have components designed to be reused in a variety of scenarios. You also need a large catalog of such components. One other key that can be easily overlooked is that you need to be able to find all of these components. Again, eBay has addressed this need by creating an Eclipse plug-in, the Component Viewer, as shown below.

The explorer shows all the components from all the projects in your Eclipse workspace, organized by project. The explorer lets you browse to find the one you want, then you can use the project dependency plug-in (see Figure 1) and adjust your project dependencies if needed.

The Components Viewer helps with finding a specific type of class used in eBay applications. eBay has built tools for the even more-general issue of finding classes — in particular, library classes. eBay engineers are definitely believers in the DRTW (Don't Reinvent The Wheel) and leverage open source libraries aggressively. This can lead to a large number of libraries and a large number of library classes. This can present a challenge in finding such classes, and once again, there is an eBay plug-in to help with this: the JAR Explorer.

These tools help developers find classes they need and provide insight using visualizations. They are all Eclipse plug-ins written by eBay to help eBay developers. However, there are many other valuable Eclipse plug-ins used by eBay that were not written by eBay. There are so many plug-ins available to Eclipse, and you can further your organization by simply identifying existing plug-ins that address a need. Let's take a look at some used by eBay.

Back to top

Leveraging other Eclipse plug-ins

Your organization is bound to have specialized needs. Eclipse provides a platform for creating plug-ins that address those needs. We have seen how this is the case for a demanding organization like eBay. Not all of your needs are going to be totally specific to you. In many cases, they may be specialized, but still common to other organizations, as well. In such cases, there is likely to already be an Eclipse plug-in addressing those. Again, we will look at how this is true for eBay, but being able to use Eclipse and its plug-in ecosystem is not unique to eBay.

Developers at eBay do many things common to developers everywhere. They need to work with application servers. In eBay's case, this is the IBM® WebSphere® Application Server Community Edition, which is essentially Apache Geronimo. Naturally, eBay leverages the Eclipse Web Tools Project (WTP) to use plug-ins like its servers views for managing and deploying to application servers, as shown below.

You might have noticed that the Megajar dialog (see Figure 6) included a message about "recycling the server." This is the server in question. The open source nature of WTP made it possible for eBay's own plug-ins to interact with WTP to create a preconfigured server instance automatically. No sense in developers wasting time on something like that. Running code on a server is a common task for any Web developer.

Another common task for any developer in any size organization is managing source code. Let's take a look at how that works for an organization as large as eBay and the role played by Eclipse.

Back to top

Integrating with source control: Rational ClearCase plug-in

Source-code management is a well-known problem, and there are numerous solutions out there. Eclipse ships with CVS support. At eBay, IBM Rational® ClearCase® is used. The EDE comes with the ClearCase plug-in installed, as shown below.

Integrating a tool like ClearCase is a big part of why you use an IDE like Eclipse. The plug-in does everything you want from a source-code management tool, making it easy to check in and check out files.

No matter what your organization's source-code management tool of choice is, chances are there is an Eclipse plug-in for it. This is one of the reasons for Eclipse's surge in popularity beyond Java development. Source control for a large system can be difficult, but integrating source-code management within Eclipse certainly helps. Another task an organization of eBay's size faces is managing consistency in source code. This is another challenging area that can be leveraged using Eclipse plug-ins.

Back to top

Enforcing coding standards: FindBugs

Static analysis is a valuable tool used by many organizations to help manage large code bases. The FindBugs open source project is a popular static analysis tool for Java code, and, of course, there is an Eclipse plug-in for it. It is used by eBay as part of acceptance of any source code. It is integrated into the EDE so developers can run it themselves and fix any problems it finds. Running it on a class, package, or project is simple.

FindBugs identifies many common sources for problems. Take a look at Listing 2.

                
import java.util.HashSet;
import java.util.Set;

public class DataStruct {

    private int id;
    private String code;
    
    public DataStruct(int id, String code){
        this.id = id;
        this.code = code;
    }
    public boolean equals(DataStruct other){
        return (this.id == other.id  && 
                this.code.equalsIgnoreCase(other.code));
    }
    public int hashCode(){
        return 1000*id + code.hashCode();
    }

    public static void main(String[] args) {
        Set<DataStruct> data = new HashSet<DataStruct>();
        data.add(new DataStruct(24, "ESX"));
        data.add(new DataStruct(24, "ESX"));
        System.out.println(data.size());
    }
}

What happens when you run the main method? You might expect it to print 1, but instead prints 2. Why is this? Running FindBugs sheds some light.

Ah, FindBugs tells us we defined an equals(DataStruct) method, but our class still uses Object.equals(Object). We wanted to override Object.equals(Object), but we messed up the method signature. This kind of bug can happen easily, but it's a lot less likely to happen at eBay because of its judicious use of FindBugs.

Back to top

Summary

We have seen how eBay has used Eclipse to address many of the challenges it faces. Many of these challenges are specific to eBay's circumstances. It is a huge development organization that is releasing new software (Web applications) all of the time. Eclipse allows eBay to develop plug-ins for addressing these unique needs. Not all of its needs are specific to eBay, and the large number of Eclipse plug-ins available to anyone helps address these more-general needs. Your organization can follow the same pattern, developing your own specialized plug-ins and leveraging existing plug-ins. These techniques can help your organization become productive by using Eclipse.

Resources

Learn

• Start this series with "Eclipse at eBay, Part 1: Tailoring Eclipse to the eBay architecture" to understand the challenges eBay programmers face.
  
  
• See how FindBugs can help you in the article "FindBugs, Part 1: Improve the quality of your code."
  
  
• Discover other static analysis plug-ins for Eclipse in the article "Automation for the people: Improving code with Eclipse."
  
  
• Learn how to incorporate FindBugs to attack whole classes of problems in your code "Java theory and practice: Testing with leverage."
  
  
• Explore the benefits of the Eclipse Web Tools Platform in "Build Web applications with Eclipse, WTP, and Derby."
  
  
• Do you use ClearCase? Read how to make the most of it in "The art of ClearCase deployment."
  
  
• Building Eclipse plug-ins is easier than you think, especially if you use templates. Learn more in "Building template with the Eclipse Plug-in Development Environment."
  
  
• Learn about making your own Eclipse plug-ins "How to develop an Eclipse plug-in."
  
  
• Doing Web development with Eclipse? You might want to read "Discover the Ajax Toolkit Framework for Eclipse."
  
  
• Jump-start your own development by creating Eclipse wizards with the article "Speed development with Eclipse wizards."
  
  
• Read about a brief history of eBay "How did eBay start?"
  
  
• Check out the "Recommended Eclipse reading list."
  
  
• Browse all the Eclipse content on developerWorks.
  
  
• New to Eclipse? Read the developerWorks article "Get started with Eclipse Platform" to learn its origin and architecture, and how to extend Eclipse with plug-ins.
  
  
• Expand your Eclipse skills by checking out IBM developerWorks' Eclipse project resources.
  
  
• To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
  
  
• Stay current with developerWorks' Technical events and webcasts.
  
  
• Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
  
  
• Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
  
  
• Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
  
  

Get products and technologies

• Check out the latest Eclipse technology downloads at IBM alphaWorks.
  
  
• Download Eclipse Platform and other projects from the Eclipse Foundation.
  
  
• Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
  
  
• Innovate your next open source development project with IBM trial software, available for download or on DVD.
  
  

Discuss

• The Eclipse Platform newsgroups should be your first stop to discuss questions regarding Eclipse. (Selecting this will launch your default Usenet news reader application and open eclipse.platform.)
  
  
• The Eclipse newsgroups has many resources for people interested in using and extending Eclipse.
  
  
• Participate in developerWorks blogs and get involved in the developerWorks community.
  
  

About the author

Michael Galpin has been developing Java software professionally since 1998. He currently works for eBay. He holds a degree in mathematics from the California Institute of Technology.

Rate this article

Comments

Back to top

Table of contents

• eBay's challenge
• Megajars
• Managing scale: Visualization and search tools
• Leveraging other Eclipse plug-ins
• Integrating with source control: Rational ClearCase plug-in
• Enforcing coding standards: FindBugs
• Summary
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Open source on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers