Identifying Slow WebSphere Startups or Admin Functions relating to INET6 Hostname Lookup

Body

If you are experiencing a very Slow WebSphere Application Server Startup and/or possibly WSAdmin command executions or Administrative Console operations are taking a very long time to complete, regardless of what product version you are currently running, the issue could be IP Address lookup and Hostname resolution related. This is contingent upon finding supporting evidence of relevant java/net/Inet6AddressImpl.getLocalHostName(Native Method) involvement in the associated Java cores, Thread dumps or System logs for the java process in question.

WebSphere defaults to the IPv6 protocol stack from version 6 refresh pack 2 v6.0.2 product releases onwards (v6.0.2.x, v6.1.x, v7.0.x, v8.0.x, v8.5.x). In some cases the existing DNS servers for a production environment are still running the older IPv4 protocol stack, which is 32 bit architected code not supporting the larger range of internet protocol addresses currently available in the INET6 specification.

This mismatch in protocol stacks directly impacts DNS lookup operations causing delays for the ip address and hostname resolution.

To verify if you are a candidate for the issue being described, simply collect a series of javacores or threaddumps using the kill -3 command, preferably spaced a few minutes apart, when Starting WebSphere or executing WSAdmin command that is performing slow and taking a long time to complete. 

Using either an ascii text editor or specialized tooling such as IBM Thread and Monitor Dump Analyzer or Wait, look for and locate the Main routine and/or Startup thread and verify the associated Code Stack contains the mentioned INET6 function call or native method at the top of the stack.

e. g. Partial stack frames extracted below from different Javacores collected from (17:01 - 17:08)  for the same JVM processID, using the IBM Thread and Monitor Dump Analyzer.  Both show similar main thread routines caught up in at java/net/Inet6AddressImpl.getLocalHostName(Native Method).

To further substantiate the issue, proceed to review your SystemOut.log hung thread warning entries beginning approximately (20) minutes later on or based upon your hung thread detection configuration policy timeout value and compare the stack frames from any WSVR0605W warning messages found with those previously viewed from the Javacores or Threaddumps previously.

Example: Partial stack frame from SystemOut.log (17:23) approximately (20+) minutes after the Javacores were initially generated from (17:01 - 17:08):

[7/22/15 17:23:25:921 CDT] 00000075 ThreadMonitor W   WSVR0605W: Thread "server.startup : 1" (000000aa) has been active for 620697 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:981)
    at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1352)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1300)
    at java.net.InetAddress.getAllByName(InetAddress.java:1229)
    at java.net.InetAddress.getAllByName(InetAddress.java:1151)
    at java.net.InetAddress.getByName(InetAddress.java:1101)

Summary:

Examining a series of relevant Javacores or Threaddumps in addition to WebSphere SystemOut logs, can help identify and reveal a Slow/Hung IPv6/INET6 DNS lookup hostname resolution issue, occurring in cases where Network DNS are running the older IPv4 communications protocol stack.  This directly impacts the WAS JVM Startup and/or WSAdmin command performance, as they both default to the newer IPv6 protocol specification in WAS v6.x, v7.x and v8.x. 

In most general cases, there are (3) JVM parameters or properties that can help quickly resolve the issue.

They can be configured or set under Application Servers -> Process Definition -> Java Virtual Machine -> Configuration tab as Generic JVM Arguments with a white space  
in-between each parameter specification starting with -D
-Dcom.ibm.cacheLocalHost=true  
-Djava.net.preferIPv4Stack=true  
-Djava.net.preferIPv6Addresses=false 

This should resolve the mentioned issue and restore proper performance.

Technical References:

• Slow performance or hang in HostName lookup
• wsadmin.sh/bat tool takes approximately 20 minutes to get the prompt
• IPv6 Can Cause Poor Java Performance
• High DNS traffic or slow performance on WebSphere Application Server for z/OS v5.1 and higher
• Parameters to improve Maximo performance using WebSphere
• Client on AIX generates excessive IPv6 DNS lookups for new connections
• Understanding, Tuning, and Testing the InetAddress Class and Cache

title image (modified) credit: (cc) Some rights reserved by ClkerFreeVectorImages