While investigating a problem with WebSphere Application Server, you may need to watch for events such as messages to the SystemOut.log and take action when they occur.
Resolving The Problem
The TrapIt.ear provides an easy way to perform actions based on events(message ids) in the WebSphere Application Server or based on time. If you need to monitor files (for example SystemOut.log, ffdcs, application or operating system logs, and the like), use the trapit shell script for UNIX and Linux systems.
TrapIt.ear uses the WebSphere Application Server JMX interface to watch for all JVM log events and perform an action when a matching message appears. TrapIt.ear can also trigger actions based on one or more timers or delays. TrapIt.ear can trigger several types of actions:
- Dumping the contents of the in-memory trace ring buffer
- Changing the runtime trace specification
- Running an operating system command
- Generating one or more javacores
- Generating a heap dump
For example, you could use TrapIt.ear to watch for WSVR0605W messages and generate a javacore when they occur. You could also use TrapIt.ear to enable tracing in the middle of the night, generate a javacore five minutes later, then stop tracing after ten minutes.
TrapIt.ear is a client EAR (Enterprise Archive) which you should run using the launchClient.sh (or launchClient.bat) command located in the profile of the server you wish to watch. To use TrapIt.ear, first download it to your system and then run it with no command-line arguments to read its usage help.
Displaying the TrapIt.ear help
sh> $WAS_HOME/profiles/AppSrv01/bin/launchClient.sh /full/path/to/TrapIt.ear
C:\> %WAS_HOME%\profiles\AppSrv02\bin\launchClient C:\full\path\to\TrapIt.ear
launchClient[.sh] /full/path/to/TrapIt.ear Hostname:Port TRIGGER1 ACTION1 [TRIGGER2 ACTION2] [TRIGGER3 ACTION3] ...
TrapIt.ear needs to connect to the address and boostrap port of a running WebSphere Application Server instance in order to run. You can find the port under the WebSphere Application Server profile by looking up the server's BOOTSTRAP_ADDRESS in the serverindex.xml file. Alternatively, you can navigate in the WebSphere Application Server administrative console to "Servers > Server Types > WebSphere Application Servers > $ServerName" and review the list of ports:
If you point TrapIt.ear at the bootstrap port of a nodeagent or a deployment manager, it will print a list of the servers known to them and let you watch any or all of those servers at the same time:
Example of choosing servers
TrapIt: Connecting on "aix1:9809"
TrapIt: 0: Server "dmgr" PID=16318472
TrapIt: 1: Server "Server1" PID=29819056
TrapIt: 2: Server "Server2" PID=12903372
TrapIt: 3: Server "nodeagent" PID=24314062
Type a comma separated list of servers to trigger from, or type 'all': 1,2
When watching for messages on systems with a lot of log traffic, it is much more efficient to point a copy of TrapIt.ear directly at each server. Otherwise, this feature can make it easy to use time-based triggers to gather traces and other data from multiple servers (for example, all servers in a cluster, or a selection of servers in a cell).
If security is enabled, TrapIt.ear will interactively prompt for a userid and password. The userid must have sufficient privilege to invoke the MBeans that TrapIt.ear uses, for example the userid which you use to log in to the administrative console and make configuration changes. More precise authority may be given declaratively; see MBean security policy for more details. The userid is masqueraded at the server on TrapIt.ear's behalf.
If you use a tool such as nohup to run TrapIt.ear, set the following properties in the properties/sas.client.props file under the same WebSphere Application Server profile from which you run launchClient.sh or launchClient.bat. You can encode your password in the file so that it won't be visible in clear text:
# RMI/IIOP user identity
Each TRIGGER on the command line can follow one of these two forms:
- Triggers when one of the strings appears in SystemOut.log. The limit counter allows you to control how many times TrapIt.ear will trigger. If you use a limit of 0, TrapIt.ear will trigger indefinitely.
- Trigger at a given time, specified by year, month, day, hour, minute and second. Use a 24-hour clock to trigger after noon.
- Trigger after a delay, given in seconds. A value of zero will trigger immediately.
Normally the delay is relative to the time when TrapIt.ear first started. If you put a plus sign in front of the seconds value, the delay is relative to the time when the previous trigger action completes.
Each ACTION on the command line can follow one of these five forms:
- Changes the runtime trace detail level to the value given. If you set the trace to a value with an asterisk, be sure to use quotation marks as in the second and third examples below.
- Generate one or more javacores, with a delay (in seconds) between each one.
- Generates a heap dump.
- Dumps the in-memory trace rung buffer to a file.
- Run an operating system command, with output going to the named file(s). If the command requires parameters, be sure to use quotation marks as in the fourth example below.
Watch for message WSVR0605W, then generate three javacores at 60 second intervals:
sh> ./launchClient.sh /full/path/to/TrapIt.ear localhost:2809 EVENT=1,WSVR0605W JAVACORE=3,60
Turn on tracing for a server and watch the log for the message JMSWMQ1107 or the string MQRC_CONNECTION_BROKEN. When either of these appears, create a heap dump, then wait ten more seconds and turn off tracing:
sh> ./launchClient.sh /scratch/TrapIt.ear localhost:2810 DELAY=0 TRACE="*=info:jmsApi=all:Messaging=all:com.ibm.mq.*=all" EVENT=1,JMSWMQ1107,MQRC_CONNECTION_BROKEN DELAY=+0 HEAPDUMP DELAY=+10 "TRACE=*=info"
Enable trace at a specific date and time, stop tracing ten minutes later, than take a javacore five minutes after that:
C:\> launchClient %TEMP%\TrapIt.ear 192.168.1.2:2810 TIME=2013,07,17,03,00,00 TRACE=WAS.messaging=all DELAY=+600 "TRACE=*=info" DELAY=+300 JAVACORE
Connect to a cell deployment manager and choose a list of target servers, then turn on tracing immediately. Wait one minute and generate a series of three javacores, each a minute apart, and finally turn off tracing after another minute:
sh> ./launchClient.sh /full/path/to/TrapIt.ear dmgr.example.com:2811 DELAY=0 TRACE=Transactions=all:WAS.j2c=all +DELAY=60 JAVACORE=3,60 DELAY=+60 "TRACE=*=info"
Watch for message HMGR0152W, then run solperf.sh against the application server's process identifier. Stop watching for the message after triggering three times.
sh> ./launchClient.sh /full/path/to/TrapIt.ear sun1.example.com:2811 EVENT=3,HMGR0152W "EXEC=/full/path/to/solperf.sh 230937,/tmp/solperf.out,/tmp/solperf.err"
The following output shows the TrapIt.ear watching for message "WMSG0019E" and triggering a javacore when the message appears. As the timestamps in the TrapIt.ear output show, after printing the message "Setting Trigger for log events [WMSG0019E]. Action: JAVACORE" the TrapIt.ear command waited for nearly six minutes before finding a match
sh> ./launchClient.sh /scratch/TrapIt.ear 192.168.1.2:2809 EVENT=1,WMSG0019E JAVACORE
IBM WebSphere Application Server, Release 7.0
Java™ EE Application Client Tool
Copyright IBM Corp., 1997-2008
WSCL0012I: Processing command line arguments.
WSCL0013I: Initializing the Java EE Application Client Environment.
WSCL0035I: Initialization of the Java EE Application Client Environment has completed.
WSCL0014I: Invoking the Application Client class com.ibm.ws.support.trapit.TrapIt
[20130731 17:37:33] TrapIt: WebSphere Application Server Log Watcher
[20130731 17:37:33] TrapIt: © Copyright IBM Corp. 2006, 2013
[20130731 17:37:33] TrapIt: Version: 18.104.22.16830731
[20130731 17:37:33] TrapIt: Connecting on "192.168.1.2:2809"
[20130731 17:37:33] TrapIt: Connected to "Server1" PID=29819056
[20130731 17:37:33] TrapIt: "Server1": Setting Trigger for log events [WMSG0019E]. Action: JAVACORE
[20130725 17:43:22] TrapIt: Server1: MSG MATCH: WMSG0019E: Unable to start MDB Listener SampleMDB, JMSDestination jms/SampleMDBTopic : com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager 'TEST.QMGR' with connection mode 'Client' and host name '192.168.1.3(1422)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
[20130731 17:43:22] TrapIt: Invoking "dumpThreads" on "JVM" for "Server1". Params=
[20130731 17:43:24] TrapIt: Disabled event listener on "Server1" for action JAVACORE
[20130731 17:43:24] TrapIt: Exiting
The source code for TrapIt.ear is included, in case you wish to extend the existing functionality of the tool. For example, to add a new action that fires once and completes, refer to:
To add a new action that fires multiple times at a regular interval, use the following class as a start point:
To add a new trigger, start with the following class:
A discussion thread for the TrapIt.ear tool is available on the IBM developerWorks site. If you have any comments or suggestions for TrapIt.ear, please submit them at:
15 June 2018