Topic
  • 16 replies
  • Latest Post - ‏2013-08-22T17:18:20Z by 0R54_Andreas_Schmidt
bjm88
bjm88
19 Posts

Pinned topic Websphere 7 publishing very slow

‏2009-12-18T22:26:52Z |
Hi,
my development group is experiencing extremely slow publish times for our web application. Below are our env details, we have already tried a number of things with no success, any help is greatly appreciated.

Dev OS: Windows XP SP3
Hardware: Dual Core Intel processors 1.8Ghz with 2GBs memory (all other apps shut down when using RAD)
IDE: RAD 7.5.4
Websphere 7.0.0.7 (latest patch applied)
Web App: using JSF 1.2, RichFaces 3.3, Facelets 1.1, and Spring 2.5.x. No database connections are being made at this time, the web app is very small with small number of projects and classes, as its still in a prototyping state.

Workspace configuration: No security enabled. Server set to "run server with resources within the workspace" and "never publish automatically" and "minimize application files copied to server". Also in admin consolse set "Run in development mode", "parallel start" and "start components as needed". Also removed the 3 test/sample web apps IBM ships websphere with. None of this has helped any as far as I can tell.

Notes: Found that publishing was actually faster before patch applied when we initially setup at 7.0.0.0. Currently publishing goes from
1% to 84% then restarts back to 0% then 50% then 100%, and only after 100% do console messages actually start coming about the web app shuting down, which takes a while, then starting up takes a great while. The longest period though is probably going from 50% to 100% with no output being logged. Overall time to publish is over 10min! It has made development extremely difficult. The only time I've found I get a quick less than 20 second publish is if I change just one .xhtml file. Any java code change triggers the very long process, as well as most other things or multiple changes to just standard jsf pages.

Possible root causes: Googling has suggested possibly something going out to the internet to validate DTD's or other xml files, not sure on this. Anotation processing was also mentioned, we don't use that at all. Also something about new security domain needed, but we have NO security enabled right now. Besides that I have no clue except WAS 7 or one of the patches has a major performance problem.....
Updated on 2010-03-22T19:28:39Z at 2010-03-22T19:28:39Z by bjm88
  • bjm88
    bjm88
    19 Posts

    Re: Websphere 7 publishing very slow

    ‏2010-01-19T18:56:13Z  
    I finally contacted IBM support about this after no response for several weeks. It turns out Websphere 7 has a major performance problem with annotation processing. If you do not use any IBM specific annotations related to your development/deployment I believe its save to use the below configuration, which will turn off IBM's annotation processing, which occurs on every server start and publish.

    (read full tech note from IBM to be sure its safe for you to configure your WAR like this)
    http://www-01.ibm.com/support/docview.wss?uid=swg1PK85322

    in your /WEB-INF/web.xml add the metadata-complete="true" to the the root web-app xml element

    <?xml version="1.0" encoding="UTF-8"?>
    <web-app id="WebApp_ID" version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    *metadata-complete="true"*>

    <display-name>MyWebApp</display-name>

    <welcome-file-list>
    <welcome-file>index.html</welcome-file>
    </welcome-file-list>
    <servlet />
    .....
    </web-app>
  • bjm88
    bjm88
    19 Posts

    Re: Websphere 7 publishing very slow

    ‏2010-01-28T15:59:51Z  
    • bjm88
    • ‏2010-01-19T18:56:13Z
    I finally contacted IBM support about this after no response for several weeks. It turns out Websphere 7 has a major performance problem with annotation processing. If you do not use any IBM specific annotations related to your development/deployment I believe its save to use the below configuration, which will turn off IBM's annotation processing, which occurs on every server start and publish.

    (read full tech note from IBM to be sure its safe for you to configure your WAR like this)
    http://www-01.ibm.com/support/docview.wss?uid=swg1PK85322

    in your /WEB-INF/web.xml add the metadata-complete="true" to the the root web-app xml element

    <?xml version="1.0" encoding="UTF-8"?>
    <web-app id="WebApp_ID" version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    *metadata-complete="true"*>

    <display-name>MyWebApp</display-name>

    <welcome-file-list>
    <welcome-file>index.html</welcome-file>
    </welcome-file-list>
    <servlet />
    .....
    </web-app>
    Note that using metadata-complete=true will cause the J2EE container (websphere) to not auto publish/setup EJB and WebServices based on annotations. If you need this, which I ended up needing, then you can still filter out what is scanned for annotations using one of the following methods:
    The amm.filter.properties file that is located in the was_home/properties directory.
    The amm.filter.properties file that is located in the profile_home/properties directory
    The manifest file of an application, META-INF/MANIFEST.MF
    The manifest of a Web or Enterprise JavaBeans (EJB) module within an application

    This can help, but I think the core problem is many users are going to need to use Annotations and the root cause is a major performance problem in scanning for them during startup/publish/deploy time. I would think IBM could do one of two things:
    1. On first startup cache them so changes and other publishes, like during iterative development in the IDE, do not need to scan again.
    2. (recommended) store this annotation deployment descriptor information during compile time in the manifest/classes/etc somewhere. Since its in code it can't change after compile, why wait until publish to scan for all this and set it up. Find it all at compile and do what you have to do during publish to use that info. I'm trying to get my IBM ticket to support lvl 3 for this, will keep subject updated when I hear back.
  • bjm88
    bjm88
    19 Posts

    Re: Websphere 7 publishing very slow

    ‏2010-02-09T16:34:55Z  
    • bjm88
    • ‏2010-01-28T15:59:51Z
    Note that using metadata-complete=true will cause the J2EE container (websphere) to not auto publish/setup EJB and WebServices based on annotations. If you need this, which I ended up needing, then you can still filter out what is scanned for annotations using one of the following methods:
    The amm.filter.properties file that is located in the was_home/properties directory.
    The amm.filter.properties file that is located in the profile_home/properties directory
    The manifest file of an application, META-INF/MANIFEST.MF
    The manifest of a Web or Enterprise JavaBeans (EJB) module within an application

    This can help, but I think the core problem is many users are going to need to use Annotations and the root cause is a major performance problem in scanning for them during startup/publish/deploy time. I would think IBM could do one of two things:
    1. On first startup cache them so changes and other publishes, like during iterative development in the IDE, do not need to scan again.
    2. (recommended) store this annotation deployment descriptor information during compile time in the manifest/classes/etc somewhere. Since its in code it can't change after compile, why wait until publish to scan for all this and set it up. Find it all at compile and do what you have to do during publish to use that info. I'm trying to get my IBM ticket to support lvl 3 for this, will keep subject updated when I hear back.
    Note you can also put the archive and package scan filters in EAR manifest or WAR manifest.
    I'd recommend using RAD editor in design mode, as manifests are very sensetive about line lengths and such. Details in link below

    Please check the properties "com.ibm.ws.amm.scan.context.filter.archives & com.ibm.ws.amm.scan.context.filter.packages" in the following websphere appserver V7 infocenter which has more details about setting up on the module level to exclude specific jars from scanning :
    http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/xrun_jvm.html
    will keep updating post as I find out more from my PMR with IBM
  • bjm88
    bjm88
    19 Posts

    Re: Websphere 7 publishing very slow

    ‏2010-03-22T19:28:39Z  
    • bjm88
    • ‏2010-02-09T16:34:55Z
    Note you can also put the archive and package scan filters in EAR manifest or WAR manifest.
    I'd recommend using RAD editor in design mode, as manifests are very sensetive about line lengths and such. Details in link below

    Please check the properties "com.ibm.ws.amm.scan.context.filter.archives & com.ibm.ws.amm.scan.context.filter.packages" in the following websphere appserver V7 infocenter which has more details about setting up on the module level to exclude specific jars from scanning :
    http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/xrun_jvm.html
    will keep updating post as I find out more from my PMR with IBM
    As my project has grown bigger and we are using more web services, our publish times again have gone way up, 5+ min. I have had my IBM ticket, PMR 31652,379,000, forwarded to the RAD group, there they asked I upgrade my RAD version. I tried but it failed, now I'm getting a new machine with 4+ gigs of RAM and I will try with a clean install.
  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-07-26T15:17:19Z  

    Unfortunately (and not acceptable) the problem is still there in WAS 8.0. Deployment times are >15 minutes in our project. I think IBM has to improve here urgently. Otherweise WAS8 simply cannot be used as a development server. It kills all productivity.

  • bpaskin
    bpaskin
    4282 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-07-26T16:07:27Z  

    Unfortunately (and not acceptable) the problem is still there in WAS 8.0. Deployment times are >15 minutes in our project. I think IBM has to improve here urgently. Otherweise WAS8 simply cannot be used as a development server. It kills all productivity.

    It depends on the amount of resources on your machines.  I have a 160MB ear file from a customer with over 300 Web Services, packaged in 7 war files in 1 ear and it published and deploys in about 1 minute.  The App Server is configured with 4 GB of memory and I have 16 GB workstation.

    Brian

  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-07-26T16:56:07Z  
    • bpaskin
    • ‏2013-07-26T16:07:27Z

    It depends on the amount of resources on your machines.  I have a 160MB ear file from a customer with over 300 Web Services, packaged in 7 war files in 1 ear and it published and deploys in about 1 minute.  The App Server is configured with 4 GB of memory and I have 16 GB workstation.

    Brian

    The project I'm dealing with has some EJB-Jars in the EAR, which are provided by a related project. They contain 41 Session beans and used to deploy on WAS 6.1 within about 3-5 minutes in the dev environment. Not fast, but still ok. They were already declared via annotation and with

     <installed facet="jst.ejb" version="3.0"/>

    in org.eclipse.wst.common.project.facet.core.xml.

    When migrating to WAS 8 they changed that to

     <installed facet="jst.ejb" version="3.1"/>

    Also they decided to add 40 beans annotated with @Singleton. I think it's kind of an architectural aberration, but they decided to use Singletons to do some static initializations where they were using a load-on-startup-servlet before on WAS 6.1.

    But that shouldn't even matter regarding deployment time. I think 81 EJBs is not an excessive number, unless Singletons are really so problematic. Now it takes up to 30 minutes for publishing, but never less than 10-15.

    I even added all non-EJB-jars to Ignore-Scanning-Archives in amm.filter.properties. No improvement.

    It has become barely usable. The amazing part is that the same Jars deploy in 45s on Liberty profile. So, it is possible to make that fast, which means something is really wrong in WAS8.

    My machine has 8 GB ram and I assigned 1,8 GB of heap to the VM. On WAS 6.1 768 MB were sufficient.

  • bpaskin
    bpaskin
    4282 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-07-26T17:38:32Z  

    The project I'm dealing with has some EJB-Jars in the EAR, which are provided by a related project. They contain 41 Session beans and used to deploy on WAS 6.1 within about 3-5 minutes in the dev environment. Not fast, but still ok. They were already declared via annotation and with

     <installed facet="jst.ejb" version="3.0"/>

    in org.eclipse.wst.common.project.facet.core.xml.

    When migrating to WAS 8 they changed that to

     <installed facet="jst.ejb" version="3.1"/>

    Also they decided to add 40 beans annotated with @Singleton. I think it's kind of an architectural aberration, but they decided to use Singletons to do some static initializations where they were using a load-on-startup-servlet before on WAS 6.1.

    But that shouldn't even matter regarding deployment time. I think 81 EJBs is not an excessive number, unless Singletons are really so problematic. Now it takes up to 30 minutes for publishing, but never less than 10-15.

    I even added all non-EJB-jars to Ignore-Scanning-Archives in amm.filter.properties. No improvement.

    It has become barely usable. The amazing part is that the same Jars deploy in 45s on Liberty profile. So, it is possible to make that fast, which means something is really wrong in WAS8.

    My machine has 8 GB ram and I assigned 1,8 GB of heap to the VM. On WAS 6.1 768 MB were sufficient.

    Hi, Is it on the publishing or the starting of the application?  I don't think it is fair either to compare a different app and different version of the App Server with with this problem.  They added countless changes and if it is happening during startup, it could be any number of reasons.  I have worked on several projects with hundreds of EJBs with no issues. 

    It is also not correct to compare Liberty to WAS.  Liberty is a Servlet engine with some added features.  It does not have to include all the JEE requirements and the user can pick and choose which features they want.  In WAS ALL JEE features must be included and started.

    You can turn off annotation scanning,  by using a JVM custom property or by setting metadata-complete=true in the ejb deployment descriptor.  The problem may be your EJBs when they are starting.  You should have seen a little improvement with amm turned off if the problem was with the scanning of other non EJB classes

    What version of RAD are you using and what version of WAS, with fixpak level?  If you are not at the latest levels it would be better for you to upgrade and test and then run some tracing to see where the time is being spent.

    Brian

  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-07-26T17:54:52Z  
    • bpaskin
    • ‏2013-07-26T17:38:32Z

    Hi, Is it on the publishing or the starting of the application?  I don't think it is fair either to compare a different app and different version of the App Server with with this problem.  They added countless changes and if it is happening during startup, it could be any number of reasons.  I have worked on several projects with hundreds of EJBs with no issues. 

    It is also not correct to compare Liberty to WAS.  Liberty is a Servlet engine with some added features.  It does not have to include all the JEE requirements and the user can pick and choose which features they want.  In WAS ALL JEE features must be included and started.

    You can turn off annotation scanning,  by using a JVM custom property or by setting metadata-complete=true in the ejb deployment descriptor.  The problem may be your EJBs when they are starting.  You should have seen a little improvement with amm turned off if the problem was with the scanning of other non EJB classes

    What version of RAD are you using and what version of WAS, with fixpak level?  If you are not at the latest levels it would be better for you to upgrade and test and then run some tracing to see where the time is being spent.

    Brian

    It hangs during publishing at 50%. Pretty much the same effect that people discuss here: http://fixunix.com/websphere/559309-slow-app-deployment-rad-7-5-7-0-1-a.html

    None of the suggestions discussed there helped. But the EAR deploys a lot faster if I only replace the EJB-Jars in the EAR with the old ones. So, the problem is with the EJBs being 3.1 instead of 3.0 and/or the singleton EJBs.

    We are using Eclipse Juno with the WebSphere plugin 9.0.0 (aka 8.5.5) and the server is WAS 8.0.0.6. I tried 8.5.5 with no improvement.

    As a side note: It keeps throwing NullPointerExceptions during startup, which clutter the whole startup log:

    [26.07.13 19:41:30:294 CEST] 0000002e annotations   E com.ibm.ws.amm.AnnotativeMetadataManagerImpl performMergeOperations CWWAM0001E: An exception occurred during annotation processing: {0}
                                     java.lang.NullPointerException
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.applyInheritedMethods(TransactionAttributeMergeAction.java:312)
     

    This is a known bug since November 2012 as stated here: http://www-01.ibm.com/support/docview.wss?uid=swg1PM78019

    It does'nt give me a good feeling that they couldn't or didn't want to fix it up to now. Perhaps it even has to do with the slow publishing, since it happens at annotation scanning.

  • bpaskin
    bpaskin
    4282 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-07-28T18:27:21Z  

    It hangs during publishing at 50%. Pretty much the same effect that people discuss here: http://fixunix.com/websphere/559309-slow-app-deployment-rad-7-5-7-0-1-a.html

    None of the suggestions discussed there helped. But the EAR deploys a lot faster if I only replace the EJB-Jars in the EAR with the old ones. So, the problem is with the EJBs being 3.1 instead of 3.0 and/or the singleton EJBs.

    We are using Eclipse Juno with the WebSphere plugin 9.0.0 (aka 8.5.5) and the server is WAS 8.0.0.6. I tried 8.5.5 with no improvement.

    As a side note: It keeps throwing NullPointerExceptions during startup, which clutter the whole startup log:

    [26.07.13 19:41:30:294 CEST] 0000002e annotations   E com.ibm.ws.amm.AnnotativeMetadataManagerImpl performMergeOperations CWWAM0001E: An exception occurred during annotation processing: {0}
                                     java.lang.NullPointerException
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.applyInheritedMethods(TransactionAttributeMergeAction.java:312)
     

    This is a known bug since November 2012 as stated here: http://www-01.ibm.com/support/docview.wss?uid=swg1PM78019

    It does'nt give me a good feeling that they couldn't or didn't want to fix it up to now. Perhaps it even has to do with the slow publishing, since it happens at annotation scanning.

    The problem has been fixed in 8007.  You can call IBM and get the fix for your problem before the Fixpak has been released on the website. 

    I wrote a small application with 10 EJB31 singletons that just do a @PostConstruct and set a variable.  The application is deployed and started on WASv8006 in less than 300ms.  The EJBs actually started in 127ms.  There is no caching setup on this WAS instance and has 512MB of memory allocated to the heap.

    Is it a particular EJB that is causing the issue?  Have you tried caching?  Are you doing anything that is holding up the start of the other EJBs while waiting for one to complete startup?

    Brian

  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-08-01T12:13:21Z  
    • bpaskin
    • ‏2013-07-28T18:27:21Z

    The problem has been fixed in 8007.  You can call IBM and get the fix for your problem before the Fixpak has been released on the website. 

    I wrote a small application with 10 EJB31 singletons that just do a @PostConstruct and set a variable.  The application is deployed and started on WASv8006 in less than 300ms.  The EJBs actually started in 127ms.  There is no caching setup on this WAS instance and has 512MB of memory allocated to the heap.

    Is it a particular EJB that is causing the issue?  Have you tried caching?  Are you doing anything that is holding up the start of the other EJBs while waiting for one to complete startup?

    Brian

    I'm really looking forward to the fix.

    Did you get your fast deployment time deploying an exported EAR or deploying from RAD into the test environment?

    That makes a huge differences. I get the 10-20mins for publishing only when I do that from eclipse to WAS using the WAS eclipse plugins (server connector etc.). When I build an EAR from the same project and upload and deploy that to the same WebSphere, publishing takes only about a minute. So, it is only slow when publishing and starting from eclipse the environment. The settings for "run server with resources within workspace" and "minimize application files copied.." don't really make a difference. Also, it doesn't make a difference if I use the standard or the developer version of WAS. Dito debug or non-debug mode.

    Interesting also that I don't see the NullPointerExceptions raised from annotation merging when I deploy the EAR using WAS console. Or perhaps I'm just looking into the wrong logs.

    I suspect there is something wrong in how the eclipse plugin interfaces with the server. Or something slows the process when WAS has to interact with eclipse.

    Further, I would exlude any code in the EJBs as the cause, because the delay happens before any code is being executed.

  • bpaskin
    bpaskin
    4282 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-08-01T13:42:26Z  

    I'm really looking forward to the fix.

    Did you get your fast deployment time deploying an exported EAR or deploying from RAD into the test environment?

    That makes a huge differences. I get the 10-20mins for publishing only when I do that from eclipse to WAS using the WAS eclipse plugins (server connector etc.). When I build an EAR from the same project and upload and deploy that to the same WebSphere, publishing takes only about a minute. So, it is only slow when publishing and starting from eclipse the environment. The settings for "run server with resources within workspace" and "minimize application files copied.." don't really make a difference. Also, it doesn't make a difference if I use the standard or the developer version of WAS. Dito debug or non-debug mode.

    Interesting also that I don't see the NullPointerExceptions raised from annotation merging when I deploy the EAR using WAS console. Or perhaps I'm just looking into the wrong logs.

    I suspect there is something wrong in how the eclipse plugin interfaces with the server. Or something slows the process when WAS has to interact with eclipse.

    Further, I would exlude any code in the EJBs as the cause, because the delay happens before any code is being executed.

    Hi, The times I listed were from RAD to WAS.  I had no issues with the times or the actual EJBs themselves.

    Brian

  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-08-01T17:52:44Z  
    • bpaskin
    • ‏2013-08-01T13:42:26Z

    Hi, The times I listed were from RAD to WAS.  I had no issues with the times or the actual EJBs themselves.

    Brian

    I tried 8.5.5 now. Same problem, but I see some occasional messages indicating long running threads:

    [01.08.13 19:38:59:955 CEST] 0000001c AlarmThreadMo W   UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 0" (00000039) to the alarm thread pool has been delayed for 10824 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
        at org.eclipse.jst.j2ee.ejb.internal.impl.MethodElementImpl.getEnterpriseBean(MethodElementImpl.java:907)
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.getMethodTransaction(TransactionAttributeMergeAction.java:236)
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.updateDeploymentDescriptor(TransactionAttributeMergeAction.java:213)
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.merge(TransactionAttributeMergeAction.java:145)
        at com.ibm.ws.amm.AnnotativeMetadataManagerImpl.performMergeOperations(AnnotativeMetadataManagerImpl.java:509)
        at com.ibm.ws.amm.AnnotativeMetadataManagerImpl.merge(AnnotativeMetadataManagerImpl.java:274)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:144)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:66)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.EJBJarFileImpl.processAnnotations(EJBJarFileImpl.java:584)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.EJBJarFileImpl.getDeploymentDescriptor(EJBJarFileImpl.java:475)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.EJBJarFileImpl.getDeploymentDescriptor(EJBJarFileImpl.java:253)
        at com.ibm.ws.amm.merge.ejb.EJBMergeAction.searchModulesWithEJBContent(EJBMergeAction.java:903)

    and later:

    [01.08.13 19:40:17:531 CEST] 00000039 AlarmThreadMo W   UTLS0009W: Alarm Thread "Non-deferrable Alarm : 0" (00000039) previously reported to be delayed has now completed.  It was active for approximately 88403 milliseconds.


    [01.08.13 19:45:09:977 CEST] 0000001b AlarmThreadMo W   UTLS0008W: The return of alarm thread "Deferrable Alarm : 1" (0000005d) to the alarm thread pool has been delayed for 18376 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
        at com.ibm.ws.amm.AnnotativeMetadataManagerImpl.merge(AnnotativeMetadataManagerImpl.java:262)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:144)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:66)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.WARFileImpl.processAnnotations(WARFileImpl.java:923)
        at com.ibm.ws.webfragmerger.WebFragMergerImpl.merge(WebFragMergerImpl.java:493)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.WARFileImpl.mergeAnnotationsAndFragments(WARFileImpl.java:869)

    and later

    [01.08.13 19:47:35:695 CEST] 0000005d AlarmThreadMo W   UTLS0009W: Alarm Thread "Deferrable Alarm : 1" (0000005d) previously reported to be delayed has now completed.  It was active for approximately 164098 milliseconds.


    Perhaps someone at IBM should look into this or advise how to figure out what exactly takes so long there.

  • bpaskin
    bpaskin
    4282 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-08-01T18:13:45Z  

    I tried 8.5.5 now. Same problem, but I see some occasional messages indicating long running threads:

    [01.08.13 19:38:59:955 CEST] 0000001c AlarmThreadMo W   UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 0" (00000039) to the alarm thread pool has been delayed for 10824 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
        at org.eclipse.jst.j2ee.ejb.internal.impl.MethodElementImpl.getEnterpriseBean(MethodElementImpl.java:907)
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.getMethodTransaction(TransactionAttributeMergeAction.java:236)
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.updateDeploymentDescriptor(TransactionAttributeMergeAction.java:213)
        at com.ibm.ws.amm.merge.ejb.TransactionAttributeMergeAction.merge(TransactionAttributeMergeAction.java:145)
        at com.ibm.ws.amm.AnnotativeMetadataManagerImpl.performMergeOperations(AnnotativeMetadataManagerImpl.java:509)
        at com.ibm.ws.amm.AnnotativeMetadataManagerImpl.merge(AnnotativeMetadataManagerImpl.java:274)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:144)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:66)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.EJBJarFileImpl.processAnnotations(EJBJarFileImpl.java:584)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.EJBJarFileImpl.getDeploymentDescriptor(EJBJarFileImpl.java:475)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.EJBJarFileImpl.getDeploymentDescriptor(EJBJarFileImpl.java:253)
        at com.ibm.ws.amm.merge.ejb.EJBMergeAction.searchModulesWithEJBContent(EJBMergeAction.java:903)

    and later:

    [01.08.13 19:40:17:531 CEST] 00000039 AlarmThreadMo W   UTLS0009W: Alarm Thread "Non-deferrable Alarm : 0" (00000039) previously reported to be delayed has now completed.  It was active for approximately 88403 milliseconds.


    [01.08.13 19:45:09:977 CEST] 0000001b AlarmThreadMo W   UTLS0008W: The return of alarm thread "Deferrable Alarm : 1" (0000005d) to the alarm thread pool has been delayed for 18376 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
        at com.ibm.ws.amm.AnnotativeMetadataManagerImpl.merge(AnnotativeMetadataManagerImpl.java:262)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:144)
        at com.ibm.ws.amm.commonarchive.AnnotationsProcessorImpl.merge(AnnotationsProcessorImpl.java:66)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.WARFileImpl.processAnnotations(WARFileImpl.java:923)
        at com.ibm.ws.webfragmerger.WebFragMergerImpl.merge(WebFragMergerImpl.java:493)
        at org.eclipse.jst.j2ee.commonarchivecore.internal.impl.WARFileImpl.mergeAnnotationsAndFragments(WARFileImpl.java:869)

    and later

    [01.08.13 19:47:35:695 CEST] 0000005d AlarmThreadMo W   UTLS0009W: Alarm Thread "Deferrable Alarm : 1" (0000005d) previously reported to be delayed has now completed.  It was active for approximately 164098 milliseconds.


    Perhaps someone at IBM should look into this or advise how to figure out what exactly takes so long there.

    You need to open a PMR for someone to really look deeply into you problem. I do not have enough information (logs, traces, etc) and a testcase from you to reproduce the issue.  Like I mentioned, I have no issue whatsoever with EJB 3.1, RAD and WAS.  So you are doing something different than my example.

    I do see a slew of fixes for annotations from 8.0.0.1 to 8.0.0.5, plus a f ew coming in 8.0.0.7.  v8.5.5.1 has a few also.

    Your best bet is to open a PMR.

    Regards,

    Brian

  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-08-01T18:18:17Z  
    • bpaskin
    • ‏2013-08-01T18:13:45Z

    You need to open a PMR for someone to really look deeply into you problem. I do not have enough information (logs, traces, etc) and a testcase from you to reproduce the issue.  Like I mentioned, I have no issue whatsoever with EJB 3.1, RAD and WAS.  So you are doing something different than my example.

    I do see a slew of fixes for annotations from 8.0.0.1 to 8.0.0.5, plus a f ew coming in 8.0.0.7.  v8.5.5.1 has a few also.

    Your best bet is to open a PMR.

    Regards,

    Brian

    We have quite a number of packages and classes here. My "intutition" tells me that is the main difference.

    I will now add all of them that don't have annotations to the package blacklist in amm.filter.properties.

  • 0R54_Andreas_Schmidt
    7 Posts

    Re: Websphere 7 publishing very slow

    ‏2013-08-22T17:18:20Z  

    We have quite a number of packages and classes here. My "intutition" tells me that is the main difference.

    I will now add all of them that don't have annotations to the package blacklist in amm.filter.properties.

    The amm.filter.properties didn't help. I cannot even say if id made any difference.

    Yesterday I updated to 8.0.0.7. It seems that server start and first deployment is faster, but a redeployment takes unacceptable 15 minutes.

    This is really annoying.