Topic
  • 11 replies
  • Latest Post - ‏2014-11-21T14:35:50Z by nlamonica
Hilgert
Hilgert
4 Posts

Pinned topic 500 errors

‏2011-01-05T12:52:18Z |
Customers using our EGL RUI application are getting 500 errors. This is what I see in the SystemOut.log of the server:

SRVE0293E: Servlet Error-restservices: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: SRVE0295E: Error reported: 500
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:637)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1177)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1159)
at com.ibm.javart.services.RestServiceServlet.write(RestServiceServlet.java:277)
at com.ibm.javart.services.RestServiceServlet.doHttp(RestServiceServlet.java:187)
at com.ibm.javart.services.RestServiceServlet.doPost(RestServiceServlet.java:104)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1583)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:870)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:175)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:863)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:182)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:455)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:384)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:272)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1550)

Also this message is reported now and then:
CHFW0019I: The Transport Channel Service has started chain HttpOutboundChain:<iSeriesIP>:9082.

I know that 500 errors are backend related. The application has run without problems for 4 months, but starting having these issues when the amount of users doubled. Memory and CPU usage on the server isn't high at all. Averages around 20% CPU and I have assigned 1536Mb of memory to the JVM, but when I leave it unassigned it only uses around 750Mb.

Does anybody have any idea?
Updated on 2011-02-04T21:14:29Z at 2011-02-04T21:14:29Z by Ortwin
  • SystemAdmin
    SystemAdmin
    6195 Posts

    Re: 500 errors

    ‏2011-01-12T16:24:10Z  
    Hi,
    Can you tell us more about your environment? What application server and version are you running, WebSphere Application Server? How many users are now accessing your application?

    Theresa, EGL User Experience
  • Hilgert
    Hilgert
    4 Posts

    Re: 500 errors

    ‏2011-01-12T17:20:05Z  
    Hi,
    Can you tell us more about your environment? What application server and version are you running, WebSphere Application Server? How many users are now accessing your application?

    Theresa, EGL User Experience
    Hi Theresa,

    We've put in a PMR at IBM. Number 86929,211,788

    Also more logging was setup and 1Gb of logging (1 hr only!!!) will be mailed to IBM tonight. It still seems like a HTTP connection pool or threadpool problem. Either on WAS (EGL app not closing connections properly) or iSeries related.

    Hilgert
  • Ortwin
    Ortwin
    204 Posts

    Re: 500 errors

    ‏2011-02-04T21:14:29Z  
    • Hilgert
    • ‏2011-01-12T17:20:05Z
    Hi Theresa,

    We've put in a PMR at IBM. Number 86929,211,788

    Also more logging was setup and 1Gb of logging (1 hr only!!!) will be mailed to IBM tonight. It still seems like a HTTP connection pool or threadpool problem. Either on WAS (EGL app not closing connections properly) or iSeries related.

    Hilgert
    Just for the record, this is the solution we found thanks to IBM:
    • The problem occured due to too many open files in Linux. A connection is also a open file to Linux.
    • The maximum number of open files can be set using: ulimit -n xxxxxx
    • When setting ulimit to 65535 this number is not picked up by the Java process running the Websphere server.
    • Setting the ulimit to 8192 solved the problem.
  • sathish.mya
    sathish.mya
    2 Posts

    Re: 500 errors

    ‏2013-12-18T10:22:58Z  
    • Ortwin
    • ‏2011-02-04T21:14:29Z
    Just for the record, this is the solution we found thanks to IBM:
    • The problem occured due to too many open files in Linux. A connection is also a open file to Linux.
    • The maximum number of open files can be set using: ulimit -n xxxxxx
    • When setting ulimit to 65535 this number is not picked up by the Java process running the Websphere server.
    • Setting the ulimit to 8192 solved the problem.

    Hi Ortwin ,

    Thanks for your information.

    I need small clarification about previous comment , did you change the ulimit value on websphere java process level or system level?

    like 

    $/proc/16897/cat limit

    Max open files            65536                65536                files

    thanks in advance 

     

     

  • Ortwin
    Ortwin
    204 Posts

    Re: 500 errors

    ‏2013-12-18T11:32:01Z  

    Hi Ortwin ,

    Thanks for your information.

    I need small clarification about previous comment , did you change the ulimit value on websphere java process level or system level?

    like 

    $/proc/16897/cat limit

    Max open files            65536                65536                files

    thanks in advance 

     

     

    We set it on system level.

    Later we we made an architectural change in our application to use a J2C connection factory on WAS for all out program calls. Before we used a jt400.jar per service project which resulted in more connections to the backend per user because the connection pooling was managed per service project in stead of application wide.

    Ortwin

  • sathish.mya
    sathish.mya
    2 Posts

    Re: 500 errors

    ‏2013-12-18T13:54:01Z  
    • Ortwin
    • ‏2013-12-18T11:32:01Z

    We set it on system level.

    Later we we made an architectural change in our application to use a J2C connection factory on WAS for all out program calls. Before we used a jt400.jar per service project which resulted in more connections to the backend per user because the connection pooling was managed per service project in stead of application wide.

    Ortwin

    Thank you so much for your quick reply .

    I'm also facing same Error in the SystemOut.log.can you please suggest me  if i will change the ulimit 65536 to 8192  will it work?.

    colud you explain more about  the scenario. 

    SRVE0293E: [Servlet Error]-[action]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: SRVE0295E: Error reported: 500at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:637)

    at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1187)
    at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1169)
    at com.motive.serviceview.serviceviewconsole.actions.AsyncServerRequestAction.execute(AsyncServerRequestAction.java:56)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
    at com.motive.serviceview.common.webapp.AbstractWebAppFilter.doFilter(AbstractWebAppFilter.java:64)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
    at com.motive.ISAAC.common.NGatePrincipalFilter.doFilter(NGatePrincipalFilter.java:73)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
    at com.motive.ISAAC.common.NGatePrincipalFilter.doFilter(NGatePrincipalFilter.java:73)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
    at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
    at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1613)

     

    Thanks in advance ..

    Regards,

    Sathish

  • Ortwin
    Ortwin
    204 Posts

    Re: 500 errors

    ‏2013-12-21T11:32:17Z  

    Thank you so much for your quick reply .

    I'm also facing same Error in the SystemOut.log.can you please suggest me  if i will change the ulimit 65536 to 8192  will it work?.

    colud you explain more about  the scenario. 

    SRVE0293E: [Servlet Error]-[action]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: SRVE0295E: Error reported: 500at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:637)

    at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1187)
    at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1169)
    at com.motive.serviceview.serviceviewconsole.actions.AsyncServerRequestAction.execute(AsyncServerRequestAction.java:56)
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
    at com.motive.serviceview.common.webapp.AbstractWebAppFilter.doFilter(AbstractWebAppFilter.java:64)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
    at com.motive.ISAAC.common.NGatePrincipalFilter.doFilter(NGatePrincipalFilter.java:73)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
    at com.motive.ISAAC.common.NGatePrincipalFilter.doFilter(NGatePrincipalFilter.java:73)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
    at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
    at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1613)

     

    Thanks in advance ..

    Regards,

    Sathish

    We resolved it by putting a ulimit -n 8192 in the startupscript for WebSphere in init.d

    Also we set limits of Sysctl for all services.sysctl -w fs.file-max=400000

    For setting up the J2C connection factory on WAS check http://pic.dhe.ibm.com/infocenter/iadthelp/v9/index.jsp?topic=%2Fcom.ibm.etools.iseries.jca.doc%2Ftopics%2Ftjcsette.html

     

    Ortwin

  • nlamonica
    nlamonica
    25 Posts

    Re: 500 errors

    ‏2014-11-18T16:43:04Z  
    • Ortwin
    • ‏2013-12-21T11:32:17Z

    We resolved it by putting a ulimit -n 8192 in the startupscript for WebSphere in init.d

    Also we set limits of Sysctl for all services.sysctl -w fs.file-max=400000

    For setting up the J2C connection factory on WAS check http://pic.dhe.ibm.com/infocenter/iadthelp/v9/index.jsp?topic=%2Fcom.ibm.etools.iseries.jca.doc%2Ftopics%2Ftjcsette.html

     

    Ortwin

    Hello,

    Sorry to be so ignorant, but where, exactly, does one find the init.d file?

     

    Thank you

  • Ortwin
    Ortwin
    204 Posts

    Re: 500 errors

    ‏2014-11-19T20:59:00Z  
    • nlamonica
    • ‏2014-11-18T16:43:04Z

    Hello,

    Sorry to be so ignorant, but where, exactly, does one find the init.d file?

     

    Thank you

    On Linux /etc/rc.d/init.d is the directory where to place the WebSphere startup script.

    The point is to execute 'ulimit -n 8192' when starting up WebSphere Application Server.

    Ortwin

  • nlamonica
    nlamonica
    25 Posts

    Re: 500 errors

    ‏2014-11-19T21:09:03Z  
    • Ortwin
    • ‏2014-11-19T20:59:00Z

    On Linux /etc/rc.d/init.d is the directory where to place the WebSphere startup script.

    The point is to execute 'ulimit -n 8192' when starting up WebSphere Application Server.

    Ortwin

    Thank you so much.  I appreciate it.

  • nlamonica
    nlamonica
    25 Posts

    Re: 500 errors

    ‏2014-11-21T14:35:50Z  
    • nlamonica
    • ‏2014-11-19T21:09:03Z

    Thank you so much.  I appreciate it.

    Where, on the system i, would one place this -ulimit -n 8192 command on application server startup?  I've looked all over for a startup script for the websphere application server but I have no idea where to find it.