Introduction

When using WebSphere® Studio Application Developer (Application Developer), users often create complex, interrelated projects, such as when they are developing a large J2EE application. These interrelated projects can cause build performance problems, which only increase as more projects are added. This article discusses several alternative ways to resolve this problem. While the article focuses on enterprise-scale J2EE applications, the concepts and approaches apply to the creation of any large application with many interrelated component projects.

Back to top

Set-up: optimized Java JVM configuration

Application Developer is written in Java^TM, which poses some configuration questions. The IBM ® and Sun Java JVMs have default configurations suitable for a wide range of Java applications. Their initial heap size is 4 MB, with a maximum equal to half of all the physical memory on your machine. Therefore, because Java applications cannot grow past the maximum heap size, they do not automatically consume a lot of memory, and can increase in size without consuming all of your memory.

If you use Application Developer to develop and test a large number of complex programs, you may need to adjust your maximum heap size. To get an estimate of the appropriate heap size:

1. Determine the size of your physical memory (for example, 640 MB).
2. Subtract the amount of memory used for your operating system (for example, 210 MB).
3. Subtract the amount of memory used for typical concurrent tasks (for example, if the machine is primarily used for Application Developer, subtract 100 MB).
4. Tip: The number of KB of memory used by your operating system and your active tasks is shown in the Windows Task Manager in the bottom right-hand window (in our example, it would say Mem Usage: 310,000 KB).

The total memory remaining will help you determine the appropriate maximum heap size. In our example, it is 640 - 310 = 330 MB.

Now the tradeoffs. If you make your maximum heap too small, you'll run out of memory when building large applications. If you make it too large (or run too many concurrent applications), then parts of Application Developer will continuously swap in and out, causing an apparent system "hang" when swapping back in. If you make your initial heap too small, you'll slow startup as memory is continuously grabbed in pieces. If you make the initial heap too large, you'll prevent unused memory from being available to other system tasks. Also, remember that the actual running Java program (Application Developer) requires memory in addition to its data heap. So consider a maximum heap of 2/3 of your available memory as calculated above, and an initial heap of about 2/3 of that maximum heap. (The maximum heap is the critical piece; the initial heap is just a startup optimization.) Also, for major applications, you might even chose to set your initial heap equal to the maximum heap (as is often recommended for server applications). In our example with 330MB of available memory, you could make your maximum heap 250 MB and your initial heap 150 MB. When Java objects are released, the memory is not freed until "garbage collection" takes place. When the heap is initially used up, garbage collection will walk through all old objects and finally free the memory for reuse, causing an apparent "hang" during this processing. It may be better to be aggressive in regular garbage collection rather than letting it build up and result in a long collection. The default minimum free starting collection is 30% and the default maximum free stopping collection is 60%, but we really don't have enough experience yet to recommend changing the defaults.

You can customize the Application Developer configuration using parameters with the startup program (wsappdev.exe). For example:

wsappdev.exe -data MyWorkspace 
  -vmargs -Xmx250m -Xms150m -Xminf0.40 -Xmaxf0.60

Back to top

Combined Application Developer plus Application Server swap-file thrashing

However, if you are also debugging server applications using the embedded WebSphere Test Environment (WTE) inside Application Developer, then that WTE also tries to to allocate 1/2 of physical memory for its JVM, and the total of the two heaps exceeds available memory and results in excessive memory swapping! Therefore, you need to specifically allocate the available memory between the WTE JVM (typically 100 MB) and the Application Developer JVM (the rest). So, in our example above with 640 MB of memory and 310 MB of operating system and other programs (leaving 330MB available, perhaps allocating 250 for heaps), you might allocate 80MB to WTE (not the default 640/2 = 320 MB it would otherwise use) and allocate the rest (250 - 80 = 170MB) to Application Developer (not the default 640/2 = 320 MB it would otherwise use):

wsappdev.exe -data MyWorkspace 
   -vmargs -Xmx170m -Xms150m

Note for Application Developer V5.1: Application Developer V5.1 uses the new j9 JVM, which seems to require either that ms==mx (for optimum performance but fixed memory consumption) or ms>=(ms*.80). So, the previous example either needs mx=180 or ms=140:

wsappdev.exe -data MyWorkspace 
   -vmargs -Xmx170m -Xms140m

To specify the WTE heap within Application Developer, go to the Server perspective Server Configuration view, and open your server instance Editor Window. Select the Environment tab and enter -Xmx100M in the Java VM Arguments window. Then save and close that server editor window. Now, our combination of Application Debugger plus WTE will effectively use 170 + 80 = 250 MB of heap, rather than allocating a combined 320 + 320 = 640 MB and thrashing with excessive swapping. A similar change is required if you run a stand-alone WebSphere Application Server outside of Application Developer (instead of using the embedded WTE).

Of course, your system should always be set up with good swap files (typically, at least the size of your physical memory, and initially set with the initial size equal to the maximum size on a previously defragmented disk), so that when swapping is needed it's reasonably efficient. And it's always a good idea to defragment the drive containing your workspace (and your other drives for that matter). With these steps, your system is all set.

Back to top

"Load when used" memory usage

If you start Application Developer, then make a trivial change to a JSP and save it, it will take quite awhile to perform and your memory usage reported by Windows Task Manager will go up by 40 MB or more. Even closing that JSP does not cause the reported memory usage to decrease. This is not a memory leak. The way Application Developer (and its base Eclipse WorkBench) works is that functions and features are loaded only when they are first used. Therefore the trivial JSP change causes the JSP compilation and validation classes to be loaded, but AutoBuild will also cause an incremental check/build/validation of all projects, which causes most of the other builders and validators and all project incremental build-state information to be loaded. Subsequent changes to code will typically be much faster and require very little extra memory (although, as noted above, the heap does not "garbage collect" right away).

Back to top

Background: AutoBuilds and Validation of dependent projects

In many projects, one or more Java files depend on Java files in another project, because they use those other Java classes or extend those other Java classes. Even worse, if one file in project A uses a file in project B, and a different file in project B uses a file in project C, and so on, then you can have many complex project dependencies. This often leads to circular dependencies between projects (even though there might not actually be any circular dependencies at the file level).

Thus, if one file changes, and if the Application Developer "AutoBuild" preference (Window => Preferences => Workbench => Perform build automatically on resource modification) is enabled, then every project may end up doing an incremental rebuild (including incremental validations). This can be quite time consuming. As well, if the "Validation" preference (projectName => Properties => Validation => Run validation when you save changes) is enabled, then various validators (HTML, EJB, MAP, SCHEMA, JSP^TM, XML, etc.) run and this can also be even more time consuming.

Most applications have sets of projects containing commonly used library functions, and sets of projects containing widely used base classes. A change to them typically causes massive rebuilds even though their interfaces probably have not changed and the dependent files really don't need to be recompiled (but the compilers and builders don't know this and therefore all dependent files end up being rebuilt).

Back to top

Solution 1: disable AutoBuilds (and Validation)

You can disable the AutoBuild preference (also disabling Validation), so that dependent files are not rebuilt. Unfortunately, neither are files you change. You must click on the project and select build (for an incremental build), rebuild (for a full build), or runValidation. Sorry, you cannot build just one file. If, when you save a file, you forget to build it, then you won't know about any compilation or validation errors, which can be annoying when you discover it later. Also, if you later re-enable the AutoBuild preference, an incremental build (and validation) is immediately done for every project in your workspace.

Back to top

Solution 2: disable Validation

You can leave the system-wide AutoBuild preference enabled, but still disable Validation. Unfortunately there is no system-wide preference for it, and you need to right-click each project (you cannot multi-select) and select Properties => Validation => Run validation automatically on resource modification and then uncheck it. (This is disabled if AutoBuild is disabled, and is disabled for project types without validation such as Java projects.)

If you have many projects, it can take a long time to disable (or re-enable) every project. A ValidationOn/Off plug-in that turns Validation on and off for all projects will soon be available (isn't the Eclipse facility for dynamic plug-ins great!) as part of an article on WebSphere Developer Domain article (see the WebSphere Studio Transition page).

Back to top

Solution 3: close inactive projects

If you have 30 (or even 100) projects in your application, you are usually only working on a few at a time. To avoid ongoing system-wide builds and validations, you can close your inactive projects. This is fine if they just depend on your active projects, but if your active projects depend on your inactive ones, you will get compile errors. And, in both cases, you will not be able to debug your application, because closed projects will make pieces of it unavailable. Basically, this approach sounds nice on the surface but is not really a practical solution.

Back to top

Solution 4: binary projects

As noted above, you are usually working on only a few of the projects in your application at a time. You want to avoid rebuilding the stable parts of your application, but still have their class files available when you rebuild or debug your active projects.

If the binary (compiled) part of each inactive project is stored in a JAR file, then the source in these projects can be removed. Actually, the process is to create the binary JAR, zip your source into a source JAR, delete the project contents, and then import the binary JAR contents back in (and probably the source JAR also, but leaving it as a JAR). The binary class files are extracted and available to other dependent projects, but there is no visible source, and therefore no rebuilds will occur within these projects. Even system-wide rebuilds will only end up rebuilding the active source projects.

Of course, if one or more of these binary projects depend on your active code and if you change an interface, then you need to reload the source back in from wherever you stored it, rebuild that project, recreate the binary JAR and source JAR, and revert back to a binary project again. This is onerous, but since library interfaces don't change very often, this approach may be workable, especially with several independent development teams where each team uses binary projects for the other teams' projects and normal source projects for their own projects.

A disadvantage is that the project no longer has visible source for use by the Debugger. However, if you zipped the source into a source JAR, then you can tell the Debugger to use it right-clicking on the project and selecting Properties => Java build path => select JAR => Attach source => Archive. This approach can optimize development for applications involving many projects. But an analogous approach is much simpler and more automated -- dependent project JARs.

Back to top

Solution 5: dependent project JARs

In the dependent project JARs approach, the binary contents of each project is packaged into a JAR (as with binary projects), but other projects just depend on that JAR instead of on the project. Therefore the project source does not need to be removed. This approach works because in Application Developer, a change in a dependent JAR does not cause an automatic rebuild of files and projects that use it. (One can argue that it should rebuild in these cases, but the current behavior is nice because it lets you break the project build dependencies.)

The requirement to re-create project JARs anytime the project is rebuilt would be time consuming, but the WebSphere Developer Domain article Developing J2EE utility JARs in WebSphere Studio Application Developer explains how to make Java projects automatically JAR their contents anytime the project is rebuilt. The article includes a downloadable plug-in (did I already mention that the Eclipse facility for dynamic plug-ins is great?).

In summary, to change from normal project dependencies to dependent project JARs, you want your J2EE Enterprise Application (EAR) project to contain the run-time modules, and the utility JAR program to create build-time JARs, Here is the procedure:

1. Create a J2EE Enterprise Application (EAR) project (you should already have it if you have any J2EE projects).
   
   
2. Download the utility JAR program and install that plug-in into Application Developer according to the instructions in the above article.
   
   
3. Pick a file system directory or create a temp project in which to build the dependent project JARs. A temp project lets you put the project under Software Configuration Management control and use it to rapidly "bootstrap" a workspace.
   
   
4. All EJB projects automatically create a JAR (projectName.jar) in the EAR, which is what we want and need for run-time dependencies.
   
   
5. For each EJB and WAR project, leave their run-time module dependencies to be other dependent projects (EjbProjectName.jar in the EAR). Within the J2EE view, right-click on project and select Edit module dependencies => Available dependent JARs, then select the dependent EjbProjectName.jar.
   
   
6. If you have a Java library JAR, it stays in the EAR as a library JAR. This leaves all run-time packaging (and run-time source debugging) the same as it always was.
   
   
7. For each Java, Web, or EJB project, enable the utility JAR builder. Right-click on the project and select Properties => Zip Creation => Enable and then select your target temp directory or project location. Recommendation: your EJB project temp JAR name should be clearly different from the EAR run-time JAR name (even though this is not absolutely required). We recommend the name projectName_temp.jar.
   
   
8. Remember that Java library JARs (only) must still go into the EAR, since they are required for run-time operation as well as build-time resolution.
   
   
9. For each EJB, WAR, and Java project, change its build-time javaBuildPath dependencies to be other dependent project JARs in the temp directory or project (or the EAR for dependent Java utility JARs) instead of other dependent projects.
   
   
10. For each project, remove the project dependencies from your build path: select Properties => Java build path and remove Projects.
    
    
11. For each project, add its dependent project JARs in the temp directory or project (or in the EAR for Java JARs): within the J2EE view, right-click on the project and select Properties => Java build path => Add libraries => Add external JARs.

Special note: If your project was created by importing an EJB JAR or an EAR or WAR, then you likely have an imported_classes.jar file in your project. If your original JAR contained source for all the contained classes, delete it. If your original JAR contained only classes, then it should not have be imported and expanded, but rather it should be imported unexpanded as a file-system "library JAR" rather than as a project. If it contains source but also contains some classes without source, then this is a mess. You cannot delete it if you need any of the classes, but the other re-creatable classes in the JAR will also remain in the imported JAR, which is on your run-time classpath and will likely be in conflict with changes and rebuilds that you perform. Therefore, the best approach is to import JARs with re-buildable source for all contained classes, and put any binary-only classes in an external JAR library. Keep things clean and simple.

Then when you change a source file in the project it will be rebuilt (assuming you have the AutoBuild preference set) and the project JAR will be rebuilt, but other projects will not be rebuilt since dependent JARs do not trigger rebuilds

All the normal projects (and their project source) are still available for debugging, and their compiled code and the non-code artifacts (J2EE deployment descriptors, and so on) are still in the project and still packaged in the EAR. So the entire application still works and is still debuggable, just the same as always. The only thing that has changed is that build-time dependencies use the project JARs to break the rebuild project dependencies.

Keep in mind that if your main library interfaces change, then you must manually initiate a rebuild-all to ensure that all the dependent projects are rebuilt (you may want to use the Validation On/Off plug-in to enable all project validation, then disable it when the rebuild is done). Otherwise, everything works automatically and seamlessly. Keep in mind that you should adjust the project build order by selecting Window =-> Preferences => Build Order => click project => Up/Down to ensure that your common library projects are rebuilt first anytime you do a rebuild-all.

Even with the advantages of using this dependent project JARs approach, you may have so many complex projects in a large application that you may also wish to globally disable validation using the Validation On/Off plug-in and then only re-enable it for a few active projects. Or you might even chose to disable both AutoBuild and validation, but most users find AutoBuild convenient for the current active projects.

Back to top

Summary

Many customers are now using Application Developer with a large number of complex interrelated projects, particularly when developing large J2EE applications. These interrelated projects can cause build performance problems, which increase as more projects are added. This article has described several ways to minimize these problems, the main one being the use of dependent project binary JARs instead of project build dependencies.

Resources

• WebSphere Studio Transition Page
  
  
• Developing J2EE utility JARs in WebSphere Studio Application Developer
  
  
• J2EE class loading demystified
  
  

About the authors

Barry Searle is the Migration Team Leader for WebSphere Studio Application Developer. He is a professional engineer who has worked at the IBM Toronto Lab for over ten years on various application development tools. Prior to that he had many years of industry experience developing command and control systems and leading complex communications development projects. You can reach Barry at searle@ca.ibm.com .

Ellen McKay is an Information Developer for IBM Toronto Lab. She writes online help and publications for WebSphere Studio Application Developer. You can reach Ellen at ecmckay@ca.ibm.com .

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Set-up: optimized Java JVM configuration
• Combined Application Developer plus Application Server swap-file thrashing
• "Load when used" memory usage
• Background: AutoBuilds and Validation of dependent projects
• Solution 1: disable AutoBuilds (and Validation)
• Solution 2: disable Validation
• Solution 3: close inactive projects
• Solution 4: binary projects
• Solution 5: dependent project JARs
• Summary
• Resources
• About the authors
• Comments

Next steps from IBM

Rational Application Developer features provide the comprehensive Integrated Development Environment (IDE) used to develop J2EE applications requiring Servlets and JavaServer Pages (JSPs).

---

• Try: Try the free Rational Application Developer, which helps developers quickly design, develop, test, analyze, and deploy high-quality Java, Javaserver pages, and J2EE applications.
• Article: Learn how you can deliver accurate, higher-quality J2EE apps with Rational Application Developer, open source JUnit and JUnitEE test frameworks.
• Tutorial: Build a Facebook application using PHP, the Spring framework, and J2EE.
• Buy: Rational Application Developer

My developerWorks community

Dig deeper into WebSphere on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers