Topic
14 replies Latest Post - ‏2012-09-24T19:37:43Z by MarchS
WebSphere Application outages
1 Post
ACCEPTED ANSWER

Pinned topic WebSphere Application server and HTTP serer going down frequently

‏2009-11-17T10:11:35Z |
Hi,

I am using Websphere 6.1 with fix pack 27 same is for IBM http server, It is observed that both the servers (WebShere Apps and HTTP) goes down frequently which is impacting end-users to process any request.

on reviewing logs i got certain error message.

Tue Nov 17 15:04:02 2009 00001e75 b75cbb90 - ERROR: /blddir/WAS61.NATV/NATV/ws/code/plugins.http/src/common/http/lib_rio.c line 884 : Read failed, rc=104
Tue Nov 17 15:04:02 2009 00001e75 b75cbb90 - ERROR: ws_server: serverSetFailoverStatus: Marking was-lichfl2Node01_lickastle2 down
Tue Nov 17 15:04:02 2009 00001e75 b75cbb90 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'was-lichfl2Node01_lickastle2'on host 'was-lichfl2'; will try another one

can any one help to isolate the problem.

Regards
Manoj Wadhwa
Updated on 2012-09-24T19:37:43Z at 2012-09-24T19:37:43Z by MarchS
  • SystemAdmin
    SystemAdmin
    3908 Posts
    ACCEPTED ANSWER

    Re: WebSphere Application server and HTTP serer going down frequently

    ‏2009-11-18T21:56:35Z  in response to WebSphere Application outages
    rc=104 is "connection reset by peer", meaning either the JVM crashed or some intermediate firewall/LB/router closed this connection abruptly. You'll need to corroborate thsi with logs in the AppServer or network ops.
  • HitzWitStix
    HitzWitStix
    1 Post
    ACCEPTED ANSWER

    Re: WebSphere Application server and HTTP serer going down frequently

    ‏2009-11-24T20:08:03Z  in response to WebSphere Application outages
    We just had a similar problem. WAS 6.1.0.25 on Windows. IHS process went to 90% CPU with these errors in the plug-in log:
    Mon Nov 23 10:22:53 2009 00000a60 00000f58 - ERROR: G:\blddir\WAS61.NATV\NATV\ws\code\plugins.http\src\common\http\lib_rio.c line 868 : Read failed, rc=420
    Mon Nov 23 15:11:34 2009 00000a60 00000f4c - ERROR: G:\blddir\WAS61.NATV\NATV\ws\code\plugins.http\src\common\http\lib_rio.c line 868 : Read failed, rc=420
    Tue Nov 24 13:48:16 2009 00000a60 00000f50 - ERROR: G:\blddir\WAS61.NATV\NATV\ws\code\plugins.http\src\common\http\lib_rio.c line 350 : Read failed, rc=420
    Tue Nov 24 14:00:28 2009 00000a60 00000f60 - ERROR: G:\blddir\WAS61.NATV\NATV\ws\code\plugins.http\src\common\http\lib_rio.c line 350 : Read failed, rc=420
    • websphere_admin@#W
      websphere_admin@#W
      5 Posts
      ACCEPTED ANSWER

      Re: WebSphere Application server and HTTP serer going down frequently

      ‏2009-12-11T20:45:26Z  in response to HitzWitStix
      Check JVM systemout & systemerr logs to see why server is crashing more frequently. If needed enable tracing and check logs.Check if JVM is restarting by itself due to nodeagent.If system resources are high an dnodeagent is not able to connect to JVM then it will restart JVM if autorestart option is selected with running status.
      For IHS server check access logs and error logs which provides sufficient information.
      • rbr
        rbr
        2 Posts
        ACCEPTED ANSWER

        Re: WebSphere Application server and HTTP serer going down frequently

        ‏2010-01-07T13:01:05Z  in response to websphere_admin@#W
        Hello,

        I have the same problem but : important information :

        I have 2 IHS in front of 2 WAS 6.1.23.

        Sometimes one IHS doesn't see the 2 was BUT the other IHS is seeing there very well ! So no problems on WAS.

        Restart IHS is useless. Only reboot (win 2003) is well !

        Here is my error :
        Wed Jan 06 11:57:27 2010 00000764 00000c6c - ERROR: E:\wasbld\WAS61.NATV\NATV\ws\code\plugins.http\src\common\http\lib_rio.c line 1022 : Write failed, rc=10053

        We see here error on E: that is a share drive to a SAN !

        And I see that above we have in another post : Error at G: \

        So it is possible that this share drive is the cause of our problems ??

        Regards,

        Roland
        • SystemAdmin
          SystemAdmin
          3908 Posts
          ACCEPTED ANSWER

          Re: WebSphere Application server and HTTP serer going down frequently

          ‏2010-01-08T12:15:24Z  in response to rbr
          rbr, you specifically need a fix for PM02250: http://www-01.ibm.com/support/docview.wss?uid=swg1PM02250&myns=swgws&mynp=OCSSEQTP&mync=R and it is likely not related to the previous posters issue
  • Vi_Raj
    Vi_Raj
    2 Posts
    ACCEPTED ANSWER

    Re: WebSphere Application server and HTTP server going down frequently

    ‏2010-01-07T17:08:53Z  in response to WebSphere Application outages
    Hello ,

    We are also using "Websphere 6.1 with fix pack 27 same is for IBM http server" - since the upgrade we have more issues : The Http plugin which used on IBM http server sees one of the node as down (though really the node is not down) and re-directs the user to the buddy server , the result is the end users are forced to log off their systems in middle of their service calls , we use sticky session and we dont use database session replication for some internal reasons.

    PLUGIN: ws_common: websphereShouldHandleRequest: Config was successfully reloaded
    ERROR: /blddir/WAS61.NATV/NATV/ws/code/plugins.http/src/common/http/lib_rio.c line 841 : Read failed, rc=420
    ERROR: ws_server: serverSetFailoverStatus: Marking 282_crm down
    ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to '282_crm'on host '282_crm'; will try another one

    Any suggestions on this.

    Thanks in adv
    Vi
    • SystemAdmin
      SystemAdmin
      3908 Posts
      ACCEPTED ANSWER

      Re: WebSphere Application server and HTTP server going down frequently

      ‏2010-01-08T12:17:57Z  in response to Vi_Raj
      VI_raj, You'd really need to provide a TRACE level http_plugin.log with several minutes of contect. This could easily be PM04650
  • Vi_Raj
    Vi_Raj
    2 Posts
    ACCEPTED ANSWER

    Re: WebSphere Application server and HTTP serer going down frequently

    ‏2010-01-11T19:45:00Z  in response to WebSphere Application outages
    Thanks a lot for feedback Eric , can we pls have more details on 'PM04650' mentioned in your thread.

    Vi
    • SystemAdmin
      SystemAdmin
      3908 Posts
      ACCEPTED ANSWER

      Re: WebSphere Application server and HTTP serer going down frequently

      ‏2010-01-12T00:29:25Z  in response to Vi_Raj
      Sorry, I had hoped it would close a little quicker and have full text available online. Essentially, when the Plugin re-uses an old backend connection and on the first read it realizes the socket has been closed (AppServers keepalive timeout has fired), it will report that in trace that it suspects it's keepalive timeout firing, but it won't short-circuit the markdown.

      So if you have TRACE level logging, you should be able to trace how much time elapses between uses of the specific stream that failed. If it's just over your AppServer keepalive timeout, which is often 30 seconds, then you probably want the fix.
      • rbr
        rbr
        2 Posts
        ACCEPTED ANSWER

        Re: WebSphere Application server and HTTP serer going down frequently

        ‏2010-01-14T14:21:01Z  in response to SystemAdmin
        Hello Eric,

        Thanks

        But i already have serveriotimeout=0 and the PM02250 is when it's>0

        regards,

        Rbr
        • WD98_Christopher_Lira
          1 Post
          ACCEPTED ANSWER

          Re: WebSphere Application server and HTTP serer going down frequently

          ‏2011-01-12T21:05:23Z  in response to rbr
          Is there a list of the plug-in "rc" codes and descriptions of them somewhere on the web? I've searched but can't find anything.

          Thanks
          • bharry
            bharry
            2 Posts
            ACCEPTED ANSWER

            Re: WebSphere Application server and HTTP serer going down frequently

            ‏2011-05-03T18:45:56Z  in response to WD98_Christopher_Lira
            Same question: I'm seeing rc = 2, rc = 10 and rc = 11 in our plugin logs, but have no idea what the difference is. Anyone have any documentation on what the return codes are?

            rc = 2 seems to be when an appserver is down...

            The 10's and 11's are infrequent, and running the plugin in trace mode for an extended period is not an option.

            Anyone?
            • satoh
              satoh
              1 Post
              ACCEPTED ANSWER

              Re: WebSphere Application server and HTTP serer going down frequently

              ‏2011-06-13T02:43:08Z  in response to bharry
              Hello team,
              I have the same question.
              I would like to know what the plug-in "rc" means.
              Are there any documents about these return codes?

              Thanks in advance.
  • MarchS
    MarchS
    1 Post
    ACCEPTED ANSWER

    Re: WebSphere Application server and HTTP serer going down frequently

    ‏2012-09-24T19:37:43Z  in response to WebSphere Application outages
    What we have found is that updating the AFD.SYS file with the "QFE" version of http://technet.microsoft.com/en-us/security/bulletin/ms12-009 has fixed our issue.

    From a command line after download of the patch. I.e. for Windows server 2003 SP2 x86 “WindowsServer2003-KB2645640-x86-ENU.exe /b:SP2QFE"