Topic
  • 6 replies
  • Latest Post - ‏2012-06-22T07:19:43Z by wf4232607
SystemAdmin
SystemAdmin
2736 Posts

Pinned topic IBM Java 1.6.0 hang in sun/misc/Unsafe.park

‏2012-05-30T07:09:06Z |
We are using JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20111111_94827 along with Websphere Application Server 7.0.0.21 on AIX 6.1

111 Threads in our Web application are getting blocked and system is running into a deadlock.

The 111 Threads are blocked due to following Thread which remains in Parked state.

Thread 11 is in Parked state (parked by Java Concurrency API)
3XMTHREADINFO "WebContainer : 11" J9VMThread:0x00000000328D9600, j9thread_t:0x0000000128865E00, java/lang/Thread:0x000000071CF56D98, state:P, prio=5
3XMTHREADINFO1 (native thread ID:0x32E0071, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at sun/misc/Unsafe.park(Native Method)
4XESTACKTRACE at java/util/concurrent/locks/LockSupport.park(LockSupport.java:184(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:848(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1189(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:197(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/apitransport/connection/c.getProxy(Bytecode PC:21(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/api/PoolingProxyManager.getProxy(Bytecode PC:21(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/apitransport/connection/c.getProxy(Bytecode PC:21(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/api/ARServerUser.getListRole(Bytecode PC:103(Compiled Code))

Thread 11 stack trace above shows that line at top of the Stack trace is
sun/misc/Unsafe.park(Native Method)

Parking a thread is internal implementation of the Java concurrency API.
Given this, we think the above analysis strongly points to possiblity of an IBM Java bug related to implementation of java/util/concurrent/locks/ReentrantLock.lock

Can someone let us know if there is such a known defect with IBM JRE 1.6.0 and if so is there a patch / workaround available for the same?

Thx and Regards
Abhijit R
Updated on 2012-06-22T07:19:43Z at 2012-06-22T07:19:43Z by wf4232607
  • Ian_Partridge
    Ian_Partridge
    23 Posts

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-05-30T08:22:08Z  
    This is unlikely to be a bug in the Java runtime.

    The application method com/bmc/arsys/apitransport/connection/c.getProxy() is calling ReentrantLock.lock()

    The Javadoc for this method states "If the lock is held by another thread then the current thread becomes disabled for thread scheduling purposes and lies dormant until the lock has been acquired". I suspect that is the situation in this case: another thread already holds the lock you are trying to acquire.
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-05-30T09:17:20Z  
    This is unlikely to be a bug in the Java runtime.

    The application method com/bmc/arsys/apitransport/connection/c.getProxy() is calling ReentrantLock.lock()

    The Javadoc for this method states "If the lock is held by another thread then the current thread becomes disabled for thread scheduling purposes and lies dormant until the lock has been acquired". I suspect that is the situation in this case: another thread already holds the lock you are trying to acquire.
    In the "LOCKS subcomponent dump routine", I do not see "Thread-11" waiting for any other Thread to release the lock.

    I am attaching the coredump file.

    Can you please take a look and check if this is an IBM Java bug?
    Can you please let me know if any work-around / fix exists for this issue?
    Thx & Regards
    • Abhijit R
  • Ian_Partridge
    Ian_Partridge
    23 Posts

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-05-30T09:58:22Z  
    In the "LOCKS subcomponent dump routine", I do not see "Thread-11" waiting for any other Thread to release the lock.

    I am attaching the coredump file.

    Can you please take a look and check if this is an IBM Java bug?
    Can you please let me know if any work-around / fix exists for this issue?
    Thx & Regards
    • Abhijit R
    I'm not sure what you mean by "Thread-11". The thread in question is "WebContainer : 11".

    Locks that are part of java/util/concurrent, such as ReentrantLock, are not listed in the LOCKS section of a javacore file, so you cannot see which thread holds the lock from looking at a javacore file.

    You can see that "WebContainer : 11" has synchronised on a String which many other threads are waiting to synchronise on:

    3LKMONOBJECT java/lang/String@0x00000006FF37CFB0/0x00000006FF37CFBC: Flat locked by "WebContainer : 11" (0x00000000328D9600), entry count 1
    3LKWAITERQ Waiting to enter:
    3LKWAITER "WebContainer : 1" (0x0000000031E13E00)
    3LKWAITER "WebContainer : 2" (0x0000000031BD1800)
    3LKWAITER "WebContainer : 3" (0x0000000032888400)
    3LKWAITER "WebContainer : 4" (0x0000000032889B00)

    This string is probably synchronised inside com/remedy/arsys/stubs/ServerLogin.getUserRolesKeyword()

    I hope this helps you debug the problem.
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-05-30T11:25:26Z  
    I'm not sure what you mean by "Thread-11". The thread in question is "WebContainer : 11".

    Locks that are part of java/util/concurrent, such as ReentrantLock, are not listed in the LOCKS section of a javacore file, so you cannot see which thread holds the lock from looking at a javacore file.

    You can see that "WebContainer : 11" has synchronised on a String which many other threads are waiting to synchronise on:

    3LKMONOBJECT java/lang/String@0x00000006FF37CFB0/0x00000006FF37CFBC: Flat locked by "WebContainer : 11" (0x00000000328D9600), entry count 1
    3LKWAITERQ Waiting to enter:
    3LKWAITER "WebContainer : 1" (0x0000000031E13E00)
    3LKWAITER "WebContainer : 2" (0x0000000031BD1800)
    3LKWAITER "WebContainer : 3" (0x0000000032888400)
    3LKWAITER "WebContainer : 4" (0x0000000032889B00)

    This string is probably synchronised inside com/remedy/arsys/stubs/ServerLogin.getUserRolesKeyword()

    I hope this helps you debug the problem.
    Ian,

    Function getUserRolesKeyword skeleton code is as follows

    public String getUserRolesKeyword(String appname) {
    ...
    String key="SessionData.UserRoles: " + getServer()+"/"+appname;
    synchronized(key.intern()) {
    ...
    try {
    RoleInfo[] roles = getListRole(appname, getUser(), getPassword()).toArray(new RoleInfo[0]);
    ...
    }
    } catch (ARException e) {
    return "";
    }
    ...
    }
    return res;
    }
    }

    There is a synchronized block on String key.intern()

    From inside the synchronized block getListRole is called and this is causing "WebContainer : 11" to go into a parked state.

    Since "WebContainer : 11" is blocked, all the other 111 Threads are also blocked as they cannot enter the synchronized block.

    Do you mean to say that we need to debug Application code to find out who has acquired the ReentrantLock? Any suggestions regarding this?

    Thx & Regards
    • Abhijit R
  • Ian_Partridge
    Ian_Partridge
    23 Posts

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-05-30T12:03:28Z  
    Ian,

    Function getUserRolesKeyword skeleton code is as follows

    public String getUserRolesKeyword(String appname) {
    ...
    String key="SessionData.UserRoles: " + getServer()+"/"+appname;
    synchronized(key.intern()) {
    ...
    try {
    RoleInfo[] roles = getListRole(appname, getUser(), getPassword()).toArray(new RoleInfo[0]);
    ...
    }
    } catch (ARException e) {
    return "";
    }
    ...
    }
    return res;
    }
    }

    There is a synchronized block on String key.intern()

    From inside the synchronized block getListRole is called and this is causing "WebContainer : 11" to go into a parked state.

    Since "WebContainer : 11" is blocked, all the other 111 Threads are also blocked as they cannot enter the synchronized block.

    Do you mean to say that we need to debug Application code to find out who has acquired the ReentrantLock? Any suggestions regarding this?

    Thx & Regards
    • Abhijit R
    Yes, that is what I meant.

    You can see that the Reentrant lock is acquired inside com/bmc/arsys/apitransport/connection/c.getProxy()

    Where else is that lock acquired? Can you trace the locking operations? Is the lock always being unlocked correctly?

    http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReentrantLock.html explains the semantics of Reentrant lock and includes some pseudo-code explaining how to use the lock safely.
  • wf4232607
    wf4232607
    1 Post

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-06-22T07:19:43Z  
    Yes, that is what I meant.

    You can see that the Reentrant lock is acquired inside com/bmc/arsys/apitransport/connection/c.getProxy()

    Where else is that lock acquired? Can you trace the locking operations? Is the lock always being unlocked correctly?

    http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReentrantLock.html explains the semantics of Reentrant lock and includes some pseudo-code explaining how to use the lock safely.
    We've experienced this same problem.
    I found sun's jdk have the same issue. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6822370