Level: Intermediate Sumit Chawla (sumitc@us.ibm.com), IBM Certified IT Architect and Technical Lead, Java Enablement, IBM
Amit Mathur (amitmat@us.ibm.com), Senior Technical Consultant and Solutions Enablement Manager, IBM
29 Mar 2004 This 5-part series provides several tips and techniques that are commonly used for tuning Java™ applications for optimum performance on AIX®. A discussion of the applicability of each tip is also provided. Using these tips, you should be able to quickly optimize the Java environment to suit your application's needs.
Introduction
There are several performance tuning tools available for IBM eServer pSeries platform running AIX. The IBM implementation of Java running on AIX also contains several tweaks, most of which are quite well documented. Despite this, the IBM support team encounters multiple instances where the performance tuning effort is affected by the disconnect between these two sets of tools.
This series of articles shows you how to use both AIX and Java tools together, to maximize the performance of your AIX-based Java applications. Part 1, "The basics", talks about prerequisites for a successful tuning effort, and provides an overview of the tools that are available to aid in such an exercise. It is strongly recommended that you read this article in its entirety, as it can save you a lot of headaches later on.
The next three articles in this series concentrate on specific areas of performance tuning. Part 2, "The Need for Speed", talks about improving execution speed and throughput. Part 3, "More is better", looks at sizing efforts, explaining how memory can be manipulated to your advantage. Part 4, "What goes in", looks at Network and Disk I/O as the targets of performance tuning. These three articles are written to facilitate quick lookup, so you are welcome to use these as quick references instead of reading from top to bottom. Each of these articles also talks about general tips and tweaks that we have seen to be useful in the field.
Part 5, "References and conclusion", concludes the series by discussing other sources of information on this topic.
Please note that this article is restricted to J2SE information only; please refer to the documentation accompanying J2EE components for J2EE-specific tuning. If you are using an application that bundles Java, you should consult the application documentation to ensure that any changes you make will not affect the application functionality.
Before you begin
Any kind of tuning effort must take into account several things, some of which are contradictory in nature. Though each such exercise is unique, there are some steps that are universal and are almost always helpful. Here is a quick checklist of things you need in place before you can look at performance tuning. If there is any step that cannot be completed before tuning starts, there should be a very good reason for it.
Move to the latest version
Refer to IBM developer kits for AIX, Java technology edition for a list of latest available version of Java on AIX. Newer versions of Java contain performance enhancements that usually make significant difference. For example, Java 1.4 has a much more powerful Garbage Collection implementation than its predecessors.
Even if you are constrained by a requirement to use a specific version of Java, you should move to the latest available Service Refresh (SR). This is a quick way to benefit from fixes made in functionality and performance. Plus, if you encounter any kind of problems, you will need to move to the latest SR before support can be provided. Another example of upgrading to latest SRs is that switches like -Xdisableexplicitgc were added to Java 1.3.1 after it had already been released, so they are not available unless you move to a specific build.
This article concentrates on Java 1.4 and Java 1.3.1, since older versions have either reached end of service or will reach there soon. For the same reason, we concentrate on AIX 5L (specifically AIX 5.1 and AIX 5.2), not AIX 4.3.3 or earlier.
You may also want to consider moving to 64-bit versions of Java if it is feasible for your application configuration. This topic is beyond the scope of current series; we restrict our discussion to 32-bit Java unless otherwise noted. Stay tuned for an article on 64-bit Java on ESDD in the near future.
Fix it before tuning it
If you are experiencing problems like crashes, memory leaks, or application hangs, you are not ready for a tuning exercise yet. This includes cases where you have disabled parts or whole of the Just-In-Time (JIT) compiler. If either JAVA_COMPILER or JITC_COMPILEOPT environment variables are specified in the environment, your application may already be facing a performance penalty (note that you can use JITC_COMPILEOPT for optimizing your application performance as well; see Part 2 for details). You should have the problems fixed, by either upgrading to the latest Service Refresh or by engaging the IBM Java Service team, before starting the performance tuning. The Java Diagnostics Guide at IBM developer kits - diagnosis documentation provides excellent suggestions on how to debug any problems encountered with IBM Java.
Verify your environment
Usually you do not need to worry about any specific environment settings (the
JIT settings mentioned in the previous section, as well as LDR_CNTRL settings
needed for heap size modification, are some notable exceptions) as the Java
launcher sets up the environment by itself. However, if your application uses
JNI, it is essential to get the environment right for the JNI component. The SDK
Guide or README accompanying each version of Java gives a list of variables that
need to be set to specific values, and a mismatch here can lead to performance
and/or functionality problems.
Is it Java you want tuned?
While it may be a good idea to look at the various tips and adjust the Java
application environment, the performance impact will be seen only if you have
eliminated other bottlenecks. For example, if Java does not figure in the
top 5 processes eating up CPU, if other processes on the system are keeping CPU usage at 100%, it is quite likely that any changes you do to Java will not have an impact.
Does it go any faster?
The word performance is multi-faceted, with trade-off decisions to be made at
each corner. To take the guesswork out of any such decision, you must be backed
by good knowledge of expected behavior. You must also have at your disposal a
well-defined mechanism for measuring the effect of any changes you make.
Regardless of the size of the tuning effort, the steps described here will make
the tuning exercise more efficient.
Know your application characteristics
Before you start tuning the application, you must understand how the
application is expected to function. Apart from broad classifications like a
Server-Client topology or a Graphical User Interface (GUI) application, you
should understand how the application code works internally to achieve what it
is trying to do. As an example, we can show you how to calculate the amount of
CPU percentage being used during a particular run, but the interpretation of the
observed numbers is unique to each application. You should be able to
distinguish between "normal" and "abnormal" observed behavior, in order to
decide what to tune.
Note that you will not need access to Java sources in order to carry out most
of the tuning described in these documents. This does not mean that you should
treat the application as a black box. Is your application designed to complete work quickly and exit, or does it
continue running for a long time (as a server for example)? Does it allocate
memory aggressively during startup, or does it start with a small footprint?
Does it carry out a lot of recycling, or does it hold on to allocated objects?
Questions like these will shape your tuning effort.
Select a good test suite
We strongly recommend having a method in place to
quantify the gains you have achieved. A prerequisite for using any of these
tools is the existence of a repeatable, verifiable and reliable test suite that
allows you to check as much of the application functionality as possible.
Remember, tweaking one aspect of runtime behavior may adversely affect another
part in some cases. Only a good test suite will allow you to understand the
trade-offs between, for example, runtime performance and memory footprint.
Establish a baseline
Before you make a single change to the system, you
should take time to establish a clear and unambiguous method to measure the
effects of any changes you make. This method can
be as simple as using the "time" command, or more elaborate, like measuring
response time using the scripts that simulate a thousand users. Either way, the
workload used to measure the effect of a performance tweak should be diverse
enough to check as many of the different scenarios as possible. In addition to
any kind of external measurements, we recommend the use of the AIX Performance PMR Data Collection Tools
(PerfPMR).
The PerfPMR tool is actually a data gathering tool
used by AIX and other service teams. It takes a snapshot of the system for a
specified amount of time, giving a clear indication of what the system is
"doing" during this time. Instead of running each AIX tool individually, you can
simply ask PerfPMR to run the tools for you. Since the method of taking these
snapshots is well-defined, you can take multiple snapshots, at various points
during the tuning cycle, to track the tuning progress. Note that in this series, we provide examples for specific tools, not PerfPMR.
Tools for the Trade
There is a rich set of tools available to monitor
all aspects of an AIX system. Table 1 provides an outline of various tools.
These tools can be used to monitor just a single process or the system as a
whole. The AIX 5L Performance Tools Handbook and Understanding IBM eServer pSeries Performance and Sizing, and their references, are good starting points to learn
about the various tools mentioned below.
Table 1: System Monitoring Tools on AIX
|
CPU
|
Memory Subsystem
|
I/O Subsystem
|
Network Subsystem
| vmstat
iostat
ps
sar
tprof | vmstat
lsps
svmon
filemon | iostat
vmstat
lsps
filemon
lvmstat | netstat
ifconfig
tcpdump |
Each of these tools is covered briefly in the
articles that follow. There are some tools that do not fall in a specific
category, so they are briefly discussed below.
topas
topas is a useful graphical interface that will give you immediate results of what is
going on in the system. When you run it without any command-line arguments, the screen looks like this:
Topas Monitor for host: aix4prt EVENTS/QUEUES FILE/TTY
Mon Apr 16 16:16:50 2001 Interval: 2 Cswitch 5984 Readch 4864
Syscall 15776 Writech 34280
Kernel 63.1 |################## | Reads 8 Rawin 0
User 36.8 |########## | Writes 2469 Ttyout 0
Wait 0.0 | | Forks 0 Igets 0
Idle 0.0 | | Execs 0 Namei 4
Runqueue 11.5 Dirblk 0
Network KBPS I-Pack O-Pack KB-In KB-Out Waitqueue 0.0
lo0 213.9 2154.2 2153.7 107.0 106.9
tr0 34.7 16.9 34.4 0.9 33.8 PAGING MEMORY
Faults 3862 Real,MB 1023
Disk Busy% KBPS TPS KB-Read KB-Writ Steals 1580 % Comp 27.0
hdisk0 0.0 0.0 0.0 0.0 0.0 PgspIn 0 % Noncomp 73.9
PgspOut 0 % Client 0.5
Name PID CPU% PgSp Owner PageIn 0
java 16684 83.6 35.1 root PageOut 0 PAGING SPACE
java 12192 12.7 86.2 root Sios 0 Size,MB 512
lrud 1032 2.7 0.0 root % Used 1.2
aixterm 19502 0.5 0.7 root NFS (calls/sec) % Free 98.7
topas 6908 0.5 0.8 root ServerV2 0
ksh 18148 0.0 0.7 root ClientV2 0 Press:
gil 1806 0.0 0.0 root ServerV3 0 "h" for help
|
The information on the bottom left
side shows the most active processes; here, java is consuming 83.6% of CPU. The
middle right area shows the total physical memory (1 GB in this case) and Paging
space (512 MB), as well as the amount being used. So you get an excellent
overview of what the system is doing in a single screen, and then you can select
the areas to concentrate based on the information being shown here.
trace
trace
captures a sequential flow of time-stamped system events. The trace is a valuable tool
for observing system and application execution. While many of the other tools provide high level statistics such as CPU and
I/O utilization, the trace facility helps expand the information as to where the
events happened, which process is responsible, when the events took place, and
how they are affecting the system. Two post processing tools that can extract
information from the trace are utld
(in AIX 4) and curt (in AIX 5). These provide statistics on CPU utilization and process/thread
activity. The third post processing tool is splat which stands for Simple Performance Lock Analysis Tool. This
tool is used to analyze lock activity in the AIX kernel and kernel extension
for simple locks.
nmon
nmon is a free software tool that gives much of the same information as
topas, but saves the information to a file in Lotus 123 and Excel format. The download site is
http://www-106.ibm.com/developerworks/eserver/articles/analyze_aix/.
The information that is collected included CPU, disk, network, adapter
statistics, kernel counters, memory and the "top" process information.
Platform-independent Java Performance Monitoring
The Java Virtual Machine Profiling Interface (JVMPI) is supported by IBM
Java, and is useful for all aspects of performance monitoring. You can either
use a third-party profiler, or use the IBM Java profiler interface, to carry out
Java performance monitoring. See the description of -Xhprof in the Java Diagnostics Guide at IBM developer kits - diagnosis documentation for more
details. You should also refer to the README/SDK Guides provided at IBM developer kits for AIX, Java technology edition for
information specific to profiling on AIX. As an example, unless you set the
environment variable AIXTHREAD_ENRUSG=ON, the thread CPU times will not be
reported during profiling. This is documented in the README/SDK Guide for all
Java versions.
Probably the most common kind of Java performance monitoring is the verbosegc
logs. More details on how to analyze verbosegc logs is available in Fine-tuning Java garbage collection performance. Though
enabling the verbosegc traces does result in minor performance hit, the advantage gained for post-mortem analysis outweighs the
performance penalty easily. Fine-tuning Java garbage collection performance mentions how to perform tuning based on
verbosegc output.
Default Behavior of Java on AIX
This section describes the settings as they are right now. These
settings may, and in most cases will, change over time. The README or SDK Guide
accompanying the SDK are always the most up-to-date references for such
settings.
Java uses the following environment settings:
- AIXTHREAD_SCOPE=S
This setting is used to ensure that each Java thread maps 1x1 to a kernel
thread. The advantage of this approach is seen in several places; a notable
example is how Java exploits Dynamic Logical Partitioning (DLPAR); when a new
CPU is added to the partition, a Java thread can be scheduled on it. This
setting should not be changed under normal circumstances.
- AIXTHREAD_COND_DEBUG, AIXTHREAD_MUTEX_DEBUG and AIXTHREAD_RWLOCK_DEBUG
These flags are used for kernel debugging purposes. These may
sometimes be set to OFF. If not, switching them off can provide a good
performance boost.
- LDR_CNTRL=MAXDATA=0x80000000
This is the default setting on Java 1.3.1, and controls how large the Java
heap can be allowed to grow. Java 1.4 decides the LDR_CNTRL
setting based on requested heap. See Getting more memory in AIX for your Java applications for details on how to manipulate this
variable.
- JAVA_COMPILER
This decides what the Just-In-Time compiler will be. The default is jitc, which points to the IBM
JIT compiler. It can be changed to jitcg for the debug version of JIT compiler,
or to NONE for switching the JIT compiler off (which in most cases is the
absolute worst thing you can do for performance).
- IBM_MIXED_MODE_THRESHOLD
This decides the number of invocations after which the JVM JIT-compiles a
method. This setting varies by platform and version; for example, it is 600 for Java 1.3.1 on AIX.
Note that none of the above settings will override an existing setting. For
example, if you change LDR_CNTRL=MAXDATA to some other value, the value
specified by you will be used, not the defaults mentioned above.
The README/SDK Guide accompanying the IBM Java SDK explains the environment
settings that any Java Native Interface (JNI) libraries must have. If you modify
any of the environment settings specified in that list, you must make sure that
the JNI libraries are built with appropriate settings as well.
Summary
This article introduced some basic steps that can be used as a
checklist for starting off the tuning effort. The next three parts examine CPU,
Memory, Network and Disk I/O tuning.
Resources
About the authors  | | | Sumit Chawla leads the Java Enablement initiative for IBM eServer (for AIX, Windows, and Linux platforms), assisting Independent Software Vendors for IBM Servers. Sumit has a Master of Science degree in Computer Science, with almost 10 years of experience in the IT industry, and is certified by IBM as an Application Architect. He is a frequent contributor to the developerWorks eServer zone. You can contact him at sumitc@us.ibm.com. |
| | | Amit Mathur works in the IBM Solutions Development group, working primarily with IBM ISVs in enablement/performance of their apps on IBM eServer platforms and providing self-sufficiency to ISVs and customers by providing education and articles on developer works. Amit has more than fourteen years' experience working in Leading software support and development in C/C++, Java and databases on UNIX and Linux platforms. He holds a Bachelor of Engineering degree in Electronics and Telecommunication from India. You can reach Amit at amitmat@us.ibm.com. |
Rate this page
| 
