Body

Monitor Hung Threads in WebSphere.

 

1.) Use  Performance Monitoring Infrastructure  (PMI) for request metrics and counters to actively monitor Thread Pool statistics.

 

Threadpool Counter definitions

Table 1. Counter definitions. The following table describes thread pool counters.

https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/rprf_datacounter9.html

 

CreateCount

DestroyCount

ActiveCount

PoolSize

PercentUsed

PercentMaxed

DeclaredthreadHungCount

ClearedThreadHangCount

ConcurrentHungThreadCount

ActiveTime

 

Use Tivoli Performance Viewer (TPV) to view these Thread Pool stats.

 

Viewing PMI data with Tivoli Performance Viewer

https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tprf_tpvperfmodules.html

 

2.) Customize the Hung Thread Detection Policy:

Which defaults to threads running > = 10 minutes duration, before writing out WSVR0605W warning messages to SystemOut*.log (WSVR0606W are the associated completion messages).

Specify a Smaller Time interval < 10 minutes, if you want early warning signs or preemptive monitoring from the SystemOut*.log files themselves.

 

Configuring the hang detection policy

https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_confighangdet.html

 

Mainly com.ibm.websphere.threadmonitor.interval and com.ibm.websphere.threadmonitor.threshold, but there are other options that can be modified regarding Dumping Javacores and setting up earlier Alarms also.

 

3.)  Generate a Javacore:

Specifically look for "THREADS subcomponent dump routine" and "Thread Details" in the Javacore*.txt dumps.

Using Integrated Solutions Console (Admin GUI) on WebSphere Traditional Full Profile, javadump on WebSphere Liberty Profile, com.ibm.websphere.threadmonitor.dump.java for Hung Thread Detection Policy, WSAdmin command line interface, kill -3 on Unix and CTRL+BREAK on Windows:

 

a.) from the Administrative Integrated Solutions Console:

    Navigate to Troubleshooting > Java dumps and cores
    Select the server(s) to collect dumps from
    Click on Java Core

b.) using javadump (server javadump <server_name>)

c) using Hung Thread Detection com.ibm.websphere.threadmonitor.dump.java which calls dumpThreads function upon WSVR0605W hung thread detection

c.) using Unix (kill -3 PID) command

d.) using Windows CTRL+BREAK

e.) using WSAdmin command line interface:

    Launch wsadmin from the bin directory of the application server profile (such as <WAS_HOME>\profiles\<PROFILE_NAME>\bin\wsadmin). Make sure you connect to the correct SOAP port for the application server or deployment manager.

    Run the following commands:

1.)  JACL

set jvm [$AdminControl completeObjectName type=JVM,process=SERVER_NAME,*]

$AdminControl invoke $jvm dumpThreads

or

2.)  JYTHON

jvm = AdminControl.completeObjectName('type=JVM,process=SERVER_NAME,*')

AdminControl.invoke(jvm, 'dumpThreads')
Delete

 

Available TimeOuts in WebSphere:

Based upon your Environment Needs, Application Types and Configuration Setup.

 

1.) There is a QueryTimeout that can be used in conjunction with SQL Statement executions and updates, set at the DataSource as a Custom Property or on the SQL Statement itself and also an older ReadTimeout specifically for Oracle.

Configuring QueryTimeout (as Custom DataSource Property or at the SQL Statement itself)

https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/tdat_querytimeout.html

 

webSphereDefaultQueryTimeout
syncQueryTimeoutWithTransactionTimeout

PM91941: THREADS HANG IN SOCKETREAD() WHEN DATABASE IS BROUGHT DOWN

WHILE STATEMENTS ARE RUNNING

http://www-01.ibm.com/support/docview.wss?uid=swg1PM91941

 

oracle.jdbc.ReadTimeout

2.) There are HTTP, MDB, ORB and Transaction Timeouts that can be implemented based upon your application architecture:

Timeout properties summary

https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/rtrb_controllingtimeout.html

 

MDBs:

 

Listener Ports

• control_region_mdb_request_timeout
• control_region_mdb_queue_timeout_percent
• server_region_mdb_stalled_thread_dump_action

Activation Specs

• control_region_wlm_dispatch_timeout
• control_region_iiop_queue_timeout_percent
• server_region_iiop_stalled_thread_dump_action

ORB:

• ORB SSL listener keep alive
• ORB SSL listener keep alive
• WLM timeout
• Request timeout

TRANSACTION:

• Total Transaction Lifetime Timeout
• Maximum transaction timeout
• transaction_recoveryTimeout

 

3.)  There are Connection and Socket Timeouts that can be set for Web Services at the HTTP transport layer:

HTTP transport custom properties for web services applications

https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/rwbs_httptransportprop.html

• com.ibm.websphere.webservices.http.connectionIdleTimeout
• com.ibm.websphere.webservices.http.connectionKeepAlive
• com.ibm.websphere.webservices.http.connectionTimeout
• com.ibm.websphere.webservices.http.SocketTimeout