Level: Intermediate Michael Galpin (mike.sr@gmail.com), Developer, eBay
18 Mar 2008 Eclipse has become the premier integrated development environment
(IDE) for Java™ developers everywhere. Eclipse is definitely not a one-size-fits-all system, though.
Its plug-in architecture allows you to add the features you need. This is even more
important in large organizations like eBay. Such organizations have very specific
needs. Eclipse makes it easy to not only tailor a solution to those needs but also to
scale that solution across a large organization. Here in Part 2 of a two-part "Eclipse
at eBay" series, we will look at some of the organizational needs of eBay and how it has
used Eclipse to fill those needs. We will concentrate on how eBay has used Eclipse, but
the beauty is that you can employ similar tactics for your organization.
This article assumes a rudimentary familiarity with Eclipse. We will talk about the
eBay Development Environment (EDE), which is based on Eclipse V3.2 and requires JDK
V5.0 or later (see Resources for links). Some familiarity with
Web development, including HTML, JavaScript, and CSS, is useful, but not required.
eBay's challenge
Software development comes in all shapes and sizes. It is a hobby for many folks. It is
a great business for doing consulting. It is also done by organizations, both very
small and very large. When you are developing software on your own, you pick the tools
you like. You write code the way you like. You keep track of your code and maybe
perform some backups periodically. As soon as you start working with others, things
become trickier. This equation only gets more complicated when the development is being
done by an organization of thousands. Luckily, Eclipse is up to this challenge. It
provides a powerful tool for standardizing around. The extensibility afforded by its
plug-in architecture makes it easy to tailor its functionality to your organization,
regardless of its size. To demonstrate, we will look at how eBay has used Eclipse to meet its needs.
The eBay organization
You have almost certainly heard of eBay. Millions of people use it to buy and sell
merchandise each year. The eBay marketplace is a highly diversified e-commerce platform
that handles a huge number of transactions each second. To accomplish this, eBay
employs thousands of software engineers who crank out millions of lines of code. This
presents several challenges, such as managing large-scale development, integrating
source-code management, and creating code consistency and quality. Let's take a look at
each of these and see how eBay has used Eclipse to address these challenges.
Large-scale development
All of those engineers produce lots of code. This code is not one-off throwaway code.
It has to be maintained by the same large organization that developed
it. That means the same code written by one developer will often be used and modified
by many other engineers. The only approach that will scale is to standardize
— standardize the code organization, standardize coding conventions, and standardize the
tools. Standardize whatever you can.
One of the first problems facing an organization with a large code base is how to
organize the code. When you have thousands of packages and classes, you can have an
incredible array of dependencies. Imagine how hard it could be just to set up a project
with the right classpath. Now imagine thousands of developers having to do this just to
get started. Lots of time could be wasted just trying to get existing code to compile
so it can be maintained or built upon.
This problem has been addressed at eBay through automation using an Eclipse plug-in.
eBay keeps track of projects and their interdependencies using a proprietary, but very
simple, XML configuration mechanism. Each file organizes the code into a project and
specifies what other projects it depends on. An example of such a file is shown in Listing 1.
Listing 1. An eBay project file
<project
type='JAR'
path='v4samplecode'
javac.source='1.5'
javac.target='1.5'
hasV4Js='true'
hasV4Css="true"
>
<src>
<dir path='src'/>
</src>
<dependencies>
<project name='GlobalEnvironment'/>
<project name='DocumentFramework'/>
<project name='DarwinTestUtils'/>
<project name='DarwinCoreUtils'/>
<project name='DarwinCoreComponents'/>
<project name='DarwinCoreApp'/>
<project name='DarwinCoreAppResources'/>
<project name='V4DeCodeContent' />
<project name='DarwinTools' />
<project name='DarwinCodeGenImpl' />
<library name='DSF-Foo' />
<library name='EbayResources'/>
<project name='EbayUrlServices'/>
<library name='Darwin'/>
<library name='DarwinTests'/>
<library name='Dsf'/>
<library name='Kernel'/>
<library name="KernelTestAPI"/>
<library name='xml'/>
<library name='commons-jxpath'/>
<library name='icesoft-sdk'/>
<library name='junit'/>
<library name='servlet-api'/>
<library name='jsp-api'/>
<library name="jakarta-taglibs-standard"/>
</dependencies>
</project>
|
This file allows the eBay plug-in to generate Eclipse projects (.project and .classpath
files) and to reference other Eclipse projects in the classpath. Obviously, these files
are extremely important, and eBay has created a plug-in for helping edit and visualize them.
Figure 1. Project dependency plug-in
The XML files are checked into source control (more on that later), then the eBay
plug-in creates all the Eclipse projects and imports them into a developer's workspace.
People can click one button, as shown in Figure 2.
Figure 2. Initiating a build
Like the tool tip suggests, the build plug-in does a lot.
Figure 3. Build plug-in info
Clicking OK creates all the Eclipse projects and builds them. An example of a
relatively small workspace is shown below.
Figure 4. Projects in workspace
You can only imagine how painful it would be if engineers had to set up all these
projects themselves, just so they could create a project. Such is the complexity in
working with a large code base. The eBay solution is no silver bullet. It is nontrivial
to create a large number of projects with interdependencies and then build all of those
projects into a deliverable. The eBay plug-in just automates all of this. However, if
you have worked with Eclipse on large projects, you know that even having everything
well organized does not solve all problems. Just having a huge number of files and
classes can become a problem. This is another challenge eBay has addressed with
the help of an Eclipse plug-in.
Megajars
As you saw above, it is easy to wind up with a project that uses a huge
number of classes from a large code base. Have you ever tried to run Eclipse with a few
hundred-thousand source files? It can put a huge strain on your system, requiring lots
of memory (ever get an out-of-memory exception from Eclipse?) and causing everything to
run extremely slowly. In such a system, it is much better to link against the compiled
code instead of the actual source code. This is done at eBay using what eBay calls Megajars.
The idea is really quite simple. Imagine your project depends a dozen or so other
projects, which have their own dependency trees. You are not going to change any of
those projects. You just depend on them to build your project. So what you want to do is
compile all of that source code into a JAR. You don't want to lose the benefits of
being able to access the JavaDoc on that code from within Eclipse, but Eclipse makes it easy
to associate source code to a JAR for that very purpose.
This is how Megajars work. You provide a set of code to build against, and the eBay
build system provides you with the corresponding Megajar. The build plug-in then builds
your workspace against the Megajar instead of the source code. Of course, this is all
done by an Eclipse plug-in. There is a Megajars view, as shown in Figure 5.
Figure 5. The Megajars view
Clicking the Megajar icon initiates the Megajar process. You get an info box telling
you exactly what is going to happen.
Figure 6. Info box
This is a one-time step that can take a while, but is worth it. It removes the strain
on your computer and makes building and deploying your system much faster. It saves
time and saves developers from the frustrations of having a slow system. It
short, it really helps with a massive code base from a code-organization standpoint.
The eBay team has created some other tools to help developers work with their large
code base. These tools use visualization to make it easier for eBay developers to find
their way around large code bases.
Managing scale: Visualization and search tools
In any software application, there are certain files/classes that have extra
significance. This is true even when the code involved has a huge numbers of classes.
The big difference is that these classes become harder to find. Fortunately, this is
the kind of problem tools can really help with.
In Part 1,
we talk about eBay's V4 architecture. This is a front-end framework developed by eBay
to address the challenges of developing large-scale Web applications. One of the
important constructs in V4 is (not too surprisingly) called an application spec. A
single project could create numerous Web pages, each with their own content, CSS,
JavaScript, etc. All of these various pages and resources can lead to a large number of
classes, but there may be a single application spec. Luckily, eBay has built a tool for
finding and managing them: The Application Explorer is shown below.
Figure 7. The eBay Application Explorer
The Application Explorer can identify the application specs in a given project quickly
and show visualizations of the meta-data for each application spec. It can be accessed
by simply right-clicking on a project in the normal Eclipse explorer.
Figure 8. V4 context menu
The application spec is a pretty high-level object in the V4 architecture. An
application could have multiple pages. Each page could be built using low-level APIs,
adding divs, spans, forms, etc. However, V4 is a component architecture, and most pages
are actually built from such components.
The reuse of components boosts eBay developer productivity. There are several keys to
component reuse. Obviously, you must have components designed to be reused in a variety
of scenarios. You also need a large catalog of such components. One other key that can
be easily overlooked is that you need to be able to find all of these components.
Again, eBay has addressed this need by creating an Eclipse plug-in, the Component
Viewer, as shown below.
Figure 9. The Component Viewer
The explorer shows all the components from all the projects in your Eclipse workspace,
organized by project. The explorer lets you browse to find
the one you want, then you can use the project dependency plug-in (see Figure 1) and adjust
your project dependencies if needed.
The Components Viewer helps with finding a specific type of class used in eBay
applications. eBay has built tools for the even more-general issue of finding classes
— in particular, library classes. eBay engineers are definitely believers
in the DRTW (Don't Reinvent The Wheel) and leverage open source libraries aggressively.
This can lead to a large number of libraries and a large number of library classes.
This can present a challenge in finding such classes, and once again, there is an eBay
plug-in to help with this: the JAR Explorer.
Figure 10. The JAR explorer
These tools help developers find classes they need and provide insight
using visualizations. They are all Eclipse plug-ins written by eBay to help eBay
developers. However, there are many other valuable Eclipse plug-ins used by eBay that
were not written by eBay. There are so many plug-ins available to Eclipse, and you can
further your organization by simply identifying existing plug-ins that address a need.
Let's take a look at some used by eBay.
Leveraging other Eclipse plug-ins
Your organization is bound to have specialized needs. Eclipse provides a platform for
creating plug-ins that address those needs. We have seen how this is the case for a
demanding organization like eBay. Not all of your needs are going to be totally specific
to you. In many cases, they may be specialized, but still common to other organizations,
as well. In such cases, there is likely to already be an Eclipse plug-in addressing those.
Again, we will look at how this is true for eBay, but being able to use Eclipse and its
plug-in ecosystem is not unique to eBay.
Developers at eBay do many things common to developers everywhere. They need
to work with application servers. In eBay's case, this is the IBM®
WebSphere® Application Server Community Edition, which is essentially Apache
Geronimo. Naturally, eBay leverages the Eclipse Web Tools Project (WTP) to use plug-ins
like its servers views for managing and deploying to application servers, as shown below.
Figure 11. Standard server view in EDE
You might have noticed that the Megajar dialog (see Figure 6) included a message about
"recycling the server." This is the server in question. The open source nature of WTP
made it possible for eBay's own plug-ins to interact with WTP to create a preconfigured
server instance automatically. No sense in developers wasting time on something like
that. Running code on a server is a common task for any Web developer.
Another common task for any developer in any size organization is managing source code. Let's take a
look at how that works for an organization as large as eBay and the role played by Eclipse.
Integrating with source control: Rational ClearCase plug-in
Source-code management is a well-known problem, and there are numerous solutions out
there. Eclipse ships with CVS support. At eBay, IBM Rational®
ClearCase® is used. The EDE comes with the ClearCase plug-in installed, as shown
below.
Figure 12. ClearCase plug-in
Integrating a tool like ClearCase is a big part of why you use an IDE like Eclipse. The
plug-in does everything you want from a source-code management tool, making it
easy to check in and check out files.
Figure 13. ClearCase plug-in context menu
No matter what your organization's source-code management tool of choice is, chances
are there is an Eclipse plug-in for it. This is one of the reasons for Eclipse's surge
in popularity beyond Java development. Source control for a large system can be
difficult, but integrating source-code management within Eclipse certainly helps.
Another task an organization of eBay's size faces is managing consistency in
source code. This is another challenging area that can be leveraged using Eclipse plug-ins.
Enforcing coding standards: FindBugs
Static analysis is a valuable tool used by many organizations to help manage large code
bases. The FindBugs open source project is a popular static analysis tool for Java code,
and, of course, there is an Eclipse plug-in for it. It is used by eBay as part of
acceptance of any source code. It is integrated into the EDE so developers can run it
themselves and fix any problems it finds. Running it on a class, package, or project is simple.
Figure 14. FindBugs
FindBugs identifies many common sources for problems. Take a look at Listing 2.
Listing 2. Buggy code
import java.util.HashSet;
import java.util.Set;
public class DataStruct {
private int id;
private String code;
public DataStruct(int id, String code){
this.id = id;
this.code = code;
}
public boolean equals(DataStruct other){
return (this.id == other.id &&
this.code.equalsIgnoreCase(other.code));
}
public int hashCode(){
return 1000*id + code.hashCode();
}
public static void main(String[] args) {
Set<DataStruct> data = new HashSet<DataStruct>();
data.add(new DataStruct(24, "ESX"));
data.add(new DataStruct(24, "ESX"));
System.out.println(data.size());
}
}
|
What happens when you run the main method? You might expect it to print 1, but instead
prints 2. Why is this? Running FindBugs sheds some light.
Figure 15. FindBugs results
Ah, FindBugs tells us we defined an equals(DataStruct)
method, but our class still uses Object.equals(Object). We
wanted to override Object.equals(Object), but we messed up
the method signature. This kind of bug can happen easily, but it's a lot less likely to
happen at eBay because of its judicious use of FindBugs.
Summary
We have seen how eBay has used Eclipse to address many of the
challenges it faces. Many of these challenges are specific to eBay's circumstances. It
is a huge development organization that is releasing new software (Web applications)
all of the time. Eclipse allows eBay to develop plug-ins for addressing these unique
needs. Not all of its needs are specific to eBay, and the large number of Eclipse
plug-ins available to anyone helps address these more-general needs. Your organization
can follow the same pattern, developing your own specialized plug-ins and leveraging
existing plug-ins. These techniques can help your organization become productive by using Eclipse.
Resources Learn
-
Start this series with "Eclipse at eBay, Part 1: Tailoring Eclipse to the eBay
architecture" to understand the challenges eBay programmers face.
-
See how FindBugs can help you in the article "FindBugs, Part 1:
Improve the quality of your code."
-
Discover other static analysis plug-ins for Eclipse in the article "Automation for the
people: Improving code with Eclipse."
-
Learn how to incorporate FindBugs to attack whole classes of problems in your code
"Java theory and
practice: Testing with leverage."
-
Explore the benefits of the Eclipse Web Tools Platform in
"Build
Web applications with Eclipse, WTP, and Derby."
-
Do you use ClearCase? Read how to make the most of it in
"The art of
ClearCase deployment."
-
Building Eclipse plug-ins is easier than you think, especially if you use templates.
Learn more in "Building template with
the Eclipse Plug-in Development Environment."
-
Learn about making your own Eclipse plug-ins "How to develop an
Eclipse plug-in."
-
Doing Web development with Eclipse? You might want to read
"Discover the
Ajax Toolkit Framework for Eclipse."
-
Jump-start your own development by creating Eclipse wizards with the article
"Speed
development with Eclipse wizards."
-
Read about a brief history of eBay "How did eBay start?"
-
Read Simon Willison's "A (Re)-Introduction to JavaScript."
-
Check out the "Recommended Eclipse reading list."
-
Browse all the Eclipse content on developerWorks.
-
New to Eclipse? Read the developerWorks article "Get started with Eclipse Platform" to learn its origin and architecture, and how to extend Eclipse with plug-ins.
-
Expand your Eclipse skills by checking out IBM developerWorks' Eclipse project resources.
-
To listen to interesting interviews and discussions for software developers, check out check out developerWorks podcasts.
-
Stay current with developerWorks' Technical events and webcasts.
-
Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
-
Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
-
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
Get products and technologies
Discuss
-
The Eclipse Platform newsgroups should be your first stop to discuss questions regarding Eclipse. (Selecting this will launch your default Usenet news reader application and open eclipse.platform.)
-
The Eclipse newsgroups has many resources for people interested in using and extending Eclipse.
-
Participate in developerWorks blogs and get involved in the developerWorks community.
About the author  | 
| | Michael Galpin has been developing Java software professionally since 1998. He currently works for eBay. He holds a degree in mathematics from the California Institute of Technology. |
Rate this page
| 
