Topic
  • 6 replies
  • Latest Post - ‏2013-05-13T16:13:20Z by bpaskin
SystemAdmin
SystemAdmin
37421 Posts

Pinned topic scalability issue on websphere7.0.0.25

‏2013-01-30T08:44:39Z |
Hi,

We have a SandyBridge 16-core machine with Red Hat Enterprise Linux Server release 6.3 and 64G Memory. Our web application is deployed within WAS7 which is installed on this machine.(HTTP Server and App server are on the same machine). While its throughput cannot be scaled up. It can only support 2000RPM(Request per Minute,it can be considered as throughput).We have a Nehalem 12-core machine with Red Hat Enterprise Linux5.4 with the same WAS setting(same JVM Heapsize,Thread pool size etc.) can support 7000RPM.

From javacore file generated on 16-core web, we can see over 50% web container threads are blocked by Waiting for Monitor Lock on java/util/Random.

Does it imply the bottleneck is on IBM WAS side? Or very large number processes machine needs some system level tuning?
Any help or suggestion are highly appreciated. Let me know if you need more info.

Thanks,
moshiel
Updated on 2013-01-30T14:21:37Z at 2013-01-30T14:21:37Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    37421 Posts

    Re: scalability issue on websphere7.0.0.25

    ‏2013-01-30T09:26:31Z  
    BTW, Web %user cpu usage is normal while %sys is around 7% which is a little bit higher than normal 5%. I don't know whether it's ralated to the issue.
  • bpaskin
    bpaskin
    5129 Posts

    Re: scalability issue on websphere7.0.0.25

    ‏2013-01-30T12:37:37Z  
    Hi,

    Different CPUs, even by the same company, have different ways of handling different events. The java.util.Random class was not thread safe until Java 5, thus it would wait on a lock. The thread dump would hold more info. However, you most likely need to tune the system and depending on the architecture of the system (hardware), the heap size of the JVM, if compressed refs are being used or not, etc. Most people do not take into account context switching, which is very expensive. Also, you are using two different versions of the operating system. Maybe something needs to be tuned at the OS level. So the only constant in your environment is WAS.

    Regards,
    Brian
  • SystemAdmin
    SystemAdmin
    37421 Posts

    Re: scalability issue on websphere7.0.0.25

    ‏2013-01-30T14:21:37Z  
    I wonder if 1 Random thread is blocked on RNG. Maybe your 32-way system is somehow slow to seed /dev/random and there's a system-wide thing you can do to keep it full.
  • moshiel
    moshiel
    1 Post

    Re: scalability issue on websphere7.0.0.25

    ‏2013-05-09T05:30:08Z  
    I wonder if 1 Random thread is blocked on RNG. Maybe your 32-way system is somehow slow to seed /dev/random and there's a system-wide thing you can do to keep it full.

    Hi, Thanks a lot for your reply.

    The web server is 64bit machine.

    This scalabiltiy issue seems related to RHEL6.3. 

    Our 8-core machine when installed RHEL5.4 and WAS 7, it could support ~5000RPM(Request per Minute,can be considered as throughput). While the same machine changed to RHEL6.3 and installed with WAS7 using the same WAS settings, it could only support 3000RPM.

    From javacore files generated on RHEL6.3, we could see over 50% web container threads are blocked by Waiting for Monitor Lock on java/util/Random.

    Since WAS is using IBM jdk and in WAS7 it's jdk1.6, so Brian, java.util.Random class should be OK.

    Maybe Like you said we shoud tune something at OS level. I don't have clues on how to follow this, Do you have any suggestions?

    P.S, for the same machine with RHEL6.3 and installed Tomcat7, using default OS settings, it could support ~6000RPM.

     

    Regards,

    Moshiel

     

  • PetrH
    PetrH
    83 Posts

    Re: scalability issue on websphere7.0.0.25

    ‏2013-05-10T16:03:32Z  
    • moshiel
    • ‏2013-05-09T05:30:08Z

    Hi, Thanks a lot for your reply.

    The web server is 64bit machine.

    This scalabiltiy issue seems related to RHEL6.3. 

    Our 8-core machine when installed RHEL5.4 and WAS 7, it could support ~5000RPM(Request per Minute,can be considered as throughput). While the same machine changed to RHEL6.3 and installed with WAS7 using the same WAS settings, it could only support 3000RPM.

    From javacore files generated on RHEL6.3, we could see over 50% web container threads are blocked by Waiting for Monitor Lock on java/util/Random.

    Since WAS is using IBM jdk and in WAS7 it's jdk1.6, so Brian, java.util.Random class should be OK.

    Maybe Like you said we shoud tune something at OS level. I don't have clues on how to follow this, Do you have any suggestions?

    P.S, for the same machine with RHEL6.3 and installed Tomcat7, using default OS settings, it could support ~6000RPM.

     

    Regards,

    Moshiel

     

    First, at least a full thread dump would be needed, not just single monitor contention line. That will also tell us which particular Java SDK release do you have there.

    Anyway you may simply perform a test after switching to another seed data source (/dev/urandom), look here for system property "-Djava.security.egd=file:/dev/urandom": http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/com.ibm.java.security.component.doc/security-component/JaasDocs/samplesecpropsfile.html

    It may be of some use or not, as there's no full thread dump available here it's hard to say much.

  • bpaskin
    bpaskin
    5129 Posts

    Re: scalability issue on websphere7.0.0.25

    ‏2013-05-13T16:13:20Z  
    • moshiel
    • ‏2013-05-09T05:30:08Z

    Hi, Thanks a lot for your reply.

    The web server is 64bit machine.

    This scalabiltiy issue seems related to RHEL6.3. 

    Our 8-core machine when installed RHEL5.4 and WAS 7, it could support ~5000RPM(Request per Minute,can be considered as throughput). While the same machine changed to RHEL6.3 and installed with WAS7 using the same WAS settings, it could only support 3000RPM.

    From javacore files generated on RHEL6.3, we could see over 50% web container threads are blocked by Waiting for Monitor Lock on java/util/Random.

    Since WAS is using IBM jdk and in WAS7 it's jdk1.6, so Brian, java.util.Random class should be OK.

    Maybe Like you said we shoud tune something at OS level. I don't have clues on how to follow this, Do you have any suggestions?

    P.S, for the same machine with RHEL6.3 and installed Tomcat7, using default OS settings, it could support ~6000RPM.

     

    Regards,

    Moshiel

     

    Replying after 4 months...

    You need to supply a full javacore for anyone to help you out.  Again, it could be anything from the heap size to the garbage collection that is taking place to an OS level tuning.  It is hard to say without any data.  

    Tomcat is not a certified JEE Full Profile or Web Profile engine.  http://www.oracle.com/technetwork/java/javaee/overview/compatibility-jsp-136984.html

     

    Brian