Topic
  • 17 replies
  • Latest Post - ‏2013-09-12T16:12:16Z by Kumar_Y
haranath
haranath
37 Posts

Pinned topic Failed to Process response headers

‏2013-09-03T18:27:57Z |

Hello,

We are getting the error(Failed to Process response headers) in our PRODUCTION region every day. At least 100 transactions are getting failed due to this error every day. Our systems  transaction flow is as follows.

Datapower- service-1  ---> Webpshere Application Server  ---> Datapower- service-2  ---> Third party vendor(xyz).

We are receiving the above mentioned error between  Datapower- service-2 & Third party vendor(xyz). To debug this issue we have enabled network trace between our systems & third party vendor(xyz). Vendor network team resources analyzed the network trace file and provided the following reason.

Vendor side has some network settings, those are

1) If the established connection is idle for some time(4seconds) then they will drop that connection.

For example we sent one request @ 1 second and it got success, connection is idle after the transaction complete.You sent  2nd request @ 6 second then vendor will send back TCP RESET which will drop the established connection.

2) Vendor also said that it is not possible to change the time values in network settings.

For you information, we increased the number of threads count on the server @ vendor side but it didn't solve the problem.

We tried to implement the re-try option in our systems but it is not acceptable by all in our group.

Can you please give some suggestions to resolve this issue?.

 

Thanks,

Haranath.

 

 

 

 

 

 

 

 

 

 

  • kenhygh
    kenhygh
    1915 Posts

    Re: Failed to Process response headers

    ‏2013-09-03T18:56:48Z  

    Ah, the old "we know our network settings are bad but we're not going to change them"

    Just a WAG, but you might try setting your user agent (on the XML Manager) to HTTP 1.0 - which doesn't support leaving the connection open, or setting its connection timeout to something lower than 4.

    Note that these settings will have an adverse affect on your performance.

  • haranath
    haranath
    37 Posts

    Re: Failed to Process response headers

    ‏2013-09-03T19:08:13Z  
    • kenhygh
    • ‏2013-09-03T18:56:48Z

    Ah, the old "we know our network settings are bad but we're not going to change them"

    Just a WAG, but you might try setting your user agent (on the XML Manager) to HTTP 1.0 - which doesn't support leaving the connection open, or setting its connection timeout to something lower than 4.

    Note that these settings will have an adverse affect on your performance.

    Thanks for your reply, Unfortunately  converting the http  version to 1.0 also didn't solve the problem for me.

  • haranath
    haranath
    37 Posts

    Re: Failed to Process response headers

    ‏2013-09-09T14:39:04Z  

    Can any one please suggest me the solution for this issue ?.

     

     

    Thanks,

    Haranath.

  • Ramsyee
    Ramsyee
    130 Posts

    Re: Failed to Process response headers

    ‏2013-09-09T21:50:52Z  
    • haranath
    • ‏2013-09-09T14:39:04Z

    Can any one please suggest me the solution for this issue ?.

     

     

    Thanks,

    Haranath.

    Hi Haranath,

    is User using is desktop(winodws) application????

    I do see the same behaviour for one my Rest Service Proxy, but in  my case, users have installed a software on desktop so whenever intially when they fire  few transactions, response is successful after some time if they fire the request I see the logs as Failed to parse response headers, so initally I could not understand the Issue after several trail and error, I figured a way.

    So whenever they make new request or use the desktop application, they have to completely close the application (like hitting the X buttons, just like shutting the window) and again opening the application and start firing the requests ( this way i did not see the error in my proxy logs)

     

    Let me know If I have confused you too much....:)

     

  • haranath
    haranath
    37 Posts

    Re: Failed to Process response headers

    ‏2013-09-10T17:34:46Z  
    • Ramsyee
    • ‏2013-09-09T21:50:52Z

    Hi Haranath,

    is User using is desktop(winodws) application????

    I do see the same behaviour for one my Rest Service Proxy, but in  my case, users have installed a software on desktop so whenever intially when they fire  few transactions, response is successful after some time if they fire the request I see the logs as Failed to parse response headers, so initally I could not understand the Issue after several trail and error, I figured a way.

    So whenever they make new request or use the desktop application, they have to completely close the application (like hitting the X buttons, just like shutting the window) and again opening the application and start firing the requests ( this way i did not see the error in my proxy logs)

     

    Let me know If I have confused you too much....:)

     

    Thanks for your reply.

    Our application is a webservice , not a desktop application.Our webservice sends a request to third party vendor(xyz) through  datapower.

    Flow of our application is as follows.

    Consumer application ----> Datapower- service-1 < ---> Webpshere Application Server  <---> Datapower- service-2  <---> Third party vendor(xyz).

    Failed to process response headers errors occurring between Datapower- service-2 & Third party vendor service.

    Let me know if you need more details about the flow.

     

    Thanks,

    Haranath.

     

     

     

     

     

     

     

     

  • dmmicale
    dmmicale
    32 Posts

    Re: Failed to Process response headers

    ‏2013-09-10T19:04:32Z  
    • haranath
    • ‏2013-09-10T17:34:46Z

    Thanks for your reply.

    Our application is a webservice , not a desktop application.Our webservice sends a request to third party vendor(xyz) through  datapower.

    Flow of our application is as follows.

    Consumer application ----> Datapower- service-1 < ---> Webpshere Application Server  <---> Datapower- service-2  <---> Third party vendor(xyz).

    Failed to process response headers errors occurring between Datapower- service-2 & Third party vendor service.

    Let me know if you need more details about the flow.

     

    Thanks,

    Haranath.

     

     

     

     

     

     

     

     

    Sounds to me like persistent connections are your problem here.  If the backend supports persistent connections and you still want them enabled then you should go into the Advanced Proxy Settings tab and reduce the 'Back Persistent Timeout' to something like 3 seconds.

    You may also have to change the service timeouts (in the Proxy Settings tab) back side timeout/front side timeout, but if it is specifically 'idle' time related to persistent connections this may not be important.

    You could also just try disabling persistent connections ( this is a radio button on the top left of the Advanced Proxy Settings in the WSP ).

    Updated on 2013-09-10T19:07:15Z at 2013-09-10T19:07:15Z by dmmicale
  • Sudarshan Hebbar
    Sudarshan Hebbar
    57 Posts

    Re: Failed to Process response headers

    ‏2013-09-11T03:45:09Z  

    Yes I agree , it is to do with the Persistent connections. Further to dmmicale solution, I would recommend to switch off the persistent connections in the advanced tab an also to add the below snippet in your UserAgent/Inject Header Policy configurations. 

    RL Matching Expression Header Name Header Value  
    * Connection Close

     

  • haranath
    haranath
    37 Posts

    Re: Failed to Process response headers

    ‏2013-09-11T16:50:13Z  

    Yes I agree , it is to do with the Persistent connections. Further to dmmicale solution, I would recommend to switch off the persistent connections in the advanced tab an also to add the below snippet in your UserAgent/Inject Header Policy configurations. 

    RL Matching Expression Header Name Header Value  
    * Connection Close

     

    Thanks  alot  dimmicale & sudarshan for your replies.

    If we disable the persistent connections completely then it tries to establish the connection for each & every request.

    Daily we get a volume of at least 200k calls to our services, also datapower box is common to 40 different services running here.I heard that too many connections also leads to the box re-boot some times.

    I tried the option given below earlier but it didn't work in my case

    Option : Persistent connections = off , HTTP Version  = 1.0

    @Sudarshan, can you please provide some more information on the header parameter that you suggested? how it behaves if we set in the services ?.

     

  • HermannSW
    HermannSW
    5814 Posts

    Re: Failed to Process response headers

    ‏2013-09-11T21:59:10Z  
    • haranath
    • ‏2013-09-11T16:50:13Z

    Thanks  alot  dimmicale & sudarshan for your replies.

    If we disable the persistent connections completely then it tries to establish the connection for each & every request.

    Daily we get a volume of at least 200k calls to our services, also datapower box is common to 40 different services running here.I heard that too many connections also leads to the box re-boot some times.

    I tried the option given below earlier but it didn't work in my case

    Option : Persistent connections = off , HTTP Version  = 1.0

    @Sudarshan, can you please provide some more information on the header parameter that you suggested? how it behaves if we set in the services ?.

     

    > I tried the option given below earlier but it didn't work in my case
    > Option : Persistent connections = off , HTTP Version  = 1.0

    Did you do this on the frontside handler?

    Can you attach a sample request packet capture?

     

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • haranath
    haranath
    37 Posts

    Re: Failed to Process response headers

    ‏2013-09-11T23:59:40Z  
    • HermannSW
    • ‏2013-09-11T21:59:10Z

    > I tried the option given below earlier but it didn't work in my case
    > Option : Persistent connections = off , HTTP Version  = 1.0

    Did you do this on the frontside handler?

    Can you attach a sample request packet capture?

     

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

     

    Hi Hermann,

    Thanks for your help.

    All our services are web service proxy objects.

    Persistent connections = off done in "Advanced Proxy settings" tab

    HTTP Version = 1.0 done in "Proxy Settings " tab.

     

     

    Thanks,

    Haranath.

     

     

  • Sudarshan Hebbar
    Sudarshan Hebbar
    57 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T03:38:21Z  
    • haranath
    • ‏2013-09-11T16:50:13Z

    Thanks  alot  dimmicale & sudarshan for your replies.

    If we disable the persistent connections completely then it tries to establish the connection for each & every request.

    Daily we get a volume of at least 200k calls to our services, also datapower box is common to 40 different services running here.I heard that too many connections also leads to the box re-boot some times.

    I tried the option given below earlier but it didn't work in my case

    Option : Persistent connections = off , HTTP Version  = 1.0

    @Sudarshan, can you please provide some more information on the header parameter that you suggested? how it behaves if we set in the services ?.

     

    The example above mentioned sends a http header to the server saying, do not keep any connections open. It is an Useragent configuaration. This can be found under Proxy Settings in WSP under xml Manager.

    Yes you are correct that too many connections may cause a problem, however, with this the TIME_WAIT will be reduced significantly in the datapower box. 

  • HermannSW
    HermannSW
    5814 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T06:28:59Z  
    • haranath
    • ‏2013-09-11T23:59:40Z

     

    Hi Hermann,

    Thanks for your help.

    All our services are web service proxy objects.

    Persistent connections = off done in "Advanced Proxy settings" tab

    HTTP Version = 1.0 done in "Proxy Settings " tab.

     

     

    Thanks,

    Haranath.

     

     

    The packet capture is missing.

    The version 1.0 in "Proxy Settings" tab does not help you, it is labeled "HTTP Version to Server" (backend).

    Please open the HTTP frontside handler of your service, and disable "HTTP 1.1" there (by default both HTTP versions are enabled).

     

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • HermannSW
    HermannSW
    5814 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T15:09:02Z  

    Hi Haranath,

    I did download the network trace file you had attached before.

    First I have to correct myself -- reading your problem description again, your 1.0 setting for backend was correct.
    I did mix up my FSH comment with another thread.


    The whole traffix is SSL encrypted, and since I am not the NSA I cannot decrypt it ... ;-)

    But I just clicked on a packet near the start and did "Follow TCP stream" which resulted in garbage.
    But in addition it identified "tcp.stream eq 5" for me.

    What I see in Wireshark is this:

    0.341150 sec start of stream
    ...
    8.264402 sec DataPower ACK to backend
    13.25838 sec Backend FIN ACK to DataPower
    13.25891 sec DataPower TLSv1 Encrypted Alert
    13.25914 sec DataPower FIN ACK to backend
    13.29984 sec Backend RST to DataPower


    Hermann.

     

  • Kumar_Y
    Kumar_Y
    380 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T15:21:28Z  
    • HermannSW
    • ‏2013-09-12T15:09:02Z

    Hi Haranath,

    I did download the network trace file you had attached before.

    First I have to correct myself -- reading your problem description again, your 1.0 setting for backend was correct.
    I did mix up my FSH comment with another thread.


    The whole traffix is SSL encrypted, and since I am not the NSA I cannot decrypt it ... ;-)

    But I just clicked on a packet near the start and did "Follow TCP stream" which resulted in garbage.
    But in addition it identified "tcp.stream eq 5" for me.

    What I see in Wireshark is this:

    0.341150 sec start of stream
    ...
    8.264402 sec DataPower ACK to backend
    13.25838 sec Backend FIN ACK to DataPower
    13.25891 sec DataPower TLSv1 Encrypted Alert
    13.25914 sec DataPower FIN ACK to backend
    13.29984 sec Backend RST to DataPower


    Hermann.

     

    Hi Hermann,

    So if the transaction is over SSL (https) we cannot see anything via packet capture?

  • HermannSW
    HermannSW
    5814 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T15:39:18Z  
    • Kumar_Y
    • ‏2013-09-12T15:21:28Z

    Hi Hermann,

    So if the transaction is over SSL (https) we cannot see anything via packet capture?

    You can see "something", but not much ;-)

    If you posses the private key needed to decrypt, Wireshark allows you to do "Follow SSL stream" and you can see everything.

  • haranath
    haranath
    37 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T15:53:02Z  

    If the established connection is idle for more than 3.5 secs then our back end is sending TCP RESET to Datapower.

    Vendor(back end team) is saying we shouldn't  use idle connection after 3.5 secs for the subsequent requests.

    We can reduce the "Back persistent timeout" parameter value, not sure at this point how it is going to impact.

    This problem is not happening lower regions to replicate the scenario.

     

    Thanks,

    Haranath

  • Kumar_Y
    Kumar_Y
    380 Posts

    Re: Failed to Process response headers

    ‏2013-09-12T16:12:16Z  
    • HermannSW
    • ‏2013-09-12T15:39:18Z

    You can see "something", but not much ;-)

    If you posses the private key needed to decrypt, Wireshark allows you to do "Follow SSL stream" and you can see everything.

    Thanks Hermann

    Updated on 2013-09-12T16:12:23Z at 2013-09-12T16:12:23Z by Kumar_Y