Topic
6 replies Latest Post - ‏2012-06-22T07:19:43Z by wf4232607
SystemAdmin
SystemAdmin
2736 Posts
ACCEPTED ANSWER

Pinned topic IBM Java 1.6.0 hang in sun/misc/Unsafe.park

‏2012-05-30T07:09:06Z |
We are using JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20111111_94827 along with Websphere Application Server 7.0.0.21 on AIX 6.1

111 Threads in our Web application are getting blocked and system is running into a deadlock.

The 111 Threads are blocked due to following Thread which remains in Parked state.

Thread 11 is in Parked state (parked by Java Concurrency API)
3XMTHREADINFO "WebContainer : 11" J9VMThread:0x00000000328D9600, j9thread_t:0x0000000128865E00, java/lang/Thread:0x000000071CF56D98, state:P, prio=5
3XMTHREADINFO1 (native thread ID:0x32E0071, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at sun/misc/Unsafe.park(Native Method)
4XESTACKTRACE at java/util/concurrent/locks/LockSupport.park(LockSupport.java:184(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:848(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1189(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:197(Compiled Code))
4XESTACKTRACE at java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/apitransport/connection/c.getProxy(Bytecode PC:21(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/api/PoolingProxyManager.getProxy(Bytecode PC:21(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/apitransport/connection/c.getProxy(Bytecode PC:21(Compiled Code))
4XESTACKTRACE at com/bmc/arsys/api/ARServerUser.getListRole(Bytecode PC:103(Compiled Code))

Thread 11 stack trace above shows that line at top of the Stack trace is
sun/misc/Unsafe.park(Native Method)

Parking a thread is internal implementation of the Java concurrency API.
Given this, we think the above analysis strongly points to possiblity of an IBM Java bug related to implementation of java/util/concurrent/locks/ReentrantLock.lock

Can someone let us know if there is such a known defect with IBM JRE 1.6.0 and if so is there a patch / workaround available for the same?

Thx and Regards
Abhijit R
Updated on 2012-06-22T07:19:43Z at 2012-06-22T07:19:43Z by wf4232607
  • Ian_Partridge
    Ian_Partridge
    19 Posts
    ACCEPTED ANSWER

    Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

    ‏2012-05-30T08:22:08Z  in response to SystemAdmin
    This is unlikely to be a bug in the Java runtime.

    The application method com/bmc/arsys/apitransport/connection/c.getProxy() is calling ReentrantLock.lock()

    The Javadoc for this method states "If the lock is held by another thread then the current thread becomes disabled for thread scheduling purposes and lies dormant until the lock has been acquired". I suspect that is the situation in this case: another thread already holds the lock you are trying to acquire.
    • SystemAdmin
      SystemAdmin
      2736 Posts
      ACCEPTED ANSWER

      Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

      ‏2012-05-30T09:17:20Z  in response to Ian_Partridge
      In the "LOCKS subcomponent dump routine", I do not see "Thread-11" waiting for any other Thread to release the lock.

      I am attaching the coredump file.

      Can you please take a look and check if this is an IBM Java bug?
      Can you please let me know if any work-around / fix exists for this issue?
      Thx & Regards
      • Abhijit R
      • Ian_Partridge
        Ian_Partridge
        19 Posts
        ACCEPTED ANSWER

        Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

        ‏2012-05-30T09:58:22Z  in response to SystemAdmin
        I'm not sure what you mean by "Thread-11". The thread in question is "WebContainer : 11".

        Locks that are part of java/util/concurrent, such as ReentrantLock, are not listed in the LOCKS section of a javacore file, so you cannot see which thread holds the lock from looking at a javacore file.

        You can see that "WebContainer : 11" has synchronised on a String which many other threads are waiting to synchronise on:

        3LKMONOBJECT java/lang/String@0x00000006FF37CFB0/0x00000006FF37CFBC: Flat locked by "WebContainer : 11" (0x00000000328D9600), entry count 1
        3LKWAITERQ Waiting to enter:
        3LKWAITER "WebContainer : 1" (0x0000000031E13E00)
        3LKWAITER "WebContainer : 2" (0x0000000031BD1800)
        3LKWAITER "WebContainer : 3" (0x0000000032888400)
        3LKWAITER "WebContainer : 4" (0x0000000032889B00)

        This string is probably synchronised inside com/remedy/arsys/stubs/ServerLogin.getUserRolesKeyword()

        I hope this helps you debug the problem.
        • SystemAdmin
          SystemAdmin
          2736 Posts
          ACCEPTED ANSWER

          Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

          ‏2012-05-30T11:25:26Z  in response to Ian_Partridge
          Ian,

          Function getUserRolesKeyword skeleton code is as follows

          public String getUserRolesKeyword(String appname) {
          ...
          String key="SessionData.UserRoles: " + getServer()+"/"+appname;
          synchronized(key.intern()) {
          ...
          try {
          RoleInfo[] roles = getListRole(appname, getUser(), getPassword()).toArray(new RoleInfo[0]);
          ...
          }
          } catch (ARException e) {
          return "";
          }
          ...
          }
          return res;
          }
          }

          There is a synchronized block on String key.intern()

          From inside the synchronized block getListRole is called and this is causing "WebContainer : 11" to go into a parked state.

          Since "WebContainer : 11" is blocked, all the other 111 Threads are also blocked as they cannot enter the synchronized block.

          Do you mean to say that we need to debug Application code to find out who has acquired the ReentrantLock? Any suggestions regarding this?

          Thx & Regards
          • Abhijit R
          • Ian_Partridge
            Ian_Partridge
            19 Posts
            ACCEPTED ANSWER

            Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

            ‏2012-05-30T12:03:28Z  in response to SystemAdmin
            Yes, that is what I meant.

            You can see that the Reentrant lock is acquired inside com/bmc/arsys/apitransport/connection/c.getProxy()

            Where else is that lock acquired? Can you trace the locking operations? Is the lock always being unlocked correctly?

            http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReentrantLock.html explains the semantics of Reentrant lock and includes some pseudo-code explaining how to use the lock safely.
            • wf4232607
              wf4232607
              1 Post
              ACCEPTED ANSWER

              Re: IBM Java 1.6.0 hang in sun/misc/Unsafe.park

              ‏2012-06-22T07:19:43Z  in response to Ian_Partridge
              We've experienced this same problem.
              I found sun's jdk have the same issue. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6822370