Topic
10 replies Latest Post - ‏2014-03-18T09:30:46Z by P Anand
SystemAdmin
SystemAdmin
6772 Posts
ACCEPTED ANSWER

Pinned topic Backend HTTPS connection error from MPGW

‏2013-01-10T07:11:44Z |
I have a mpgw with a http FSH. I am using a dynamic backend and setting it to a https url in the request rule. I configured my user agent with the SSL proxy profile. When I run the transaction it fails. I can see in the logs that the cert validation went fine, but just after that I see this error - The header (N/A) is wrong when connecting to URL. Anyone run into this before?

I ran multiple configs with different http header settings to the backend, but of no help. I replaced the backend call with a url-open() in a stylesheet(with a skip backend). The issue remains the same.

20130110T053823Z mpgwinfo mpgw(mpgwTest): tid(667649)http://1.2.3.4: Using Backside Server: https://abcd.efgh.gateway:8443/abc?def=123
20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)http://1.2.3.4: The header (N/A) is wrong when connecting to URL 'https://abcd.efgh.gateway:8443/abc?def=123
'.
20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)http://1.2.3.4: Error occurred when connecting to URL 'https://abcd.efgh.gateway:8443/abc?def=123
'
20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)http://1.2.3.4: Backside header ('N/A') failed to parse due to: Failed to process response headers, URL: https://abcd.efgh.gateway:8443/abc?def=123
20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)errorhttp://1.2.3.4: Failed to process response headers
20130110T053823Z mpgwnotice stylepolicy(mpgwTest_policy): tid(667649)http://1.2.3.4: No error rule is matched.
Nikhil
Updated on 2013-01-16T10:31:24Z at 2013-01-16T10:31:24Z by SystemAdmin
  • swlinn
    swlinn
    1344 Posts
    ACCEPTED ANSWER

    Re: Backend HTTPS connection error from MPGW

    ‏2013-01-10T14:34:24Z  in response to SystemAdmin
    Hi Nikhil,

    Without seeing your entire log for the transaction I'd just be guessing, so a few questions.

    1) How long after the valcred log record do you see the failure? I've seen some cases where the backend just doesn't respond, which could explain these logs.
    2) Are there any logs on the backend? I've seen some cases where the backend doesn't like your request and sends a TCP RST and closes the connection, which could explain these logs.

    Regards,
    Steve
    • SystemAdmin
      SystemAdmin
      6772 Posts
      ACCEPTED ANSWER

      Re: Backend HTTPS connection error from MPGW

      ‏2013-01-10T18:28:01Z  in response to swlinn
      Thanks,
      Here are the answers -

      1) How long after the valcred log record do you see the failure? I've seen some cases where the backend just doesn't respond, which could explain these logs.
      A> Instantaneously. The timestamps are same most of the time.

      2> Are there any logs on the backend? I've seen some cases where the backend doesn't like your request and sends a TCP RST and closes the connection, which could explain these logs.
      A> Can't see any logs on the backend system. They claim that the requests are reaching them, but I doubt it based on datapower logs.

      What is confusing is that it complains about an non-existent header. What possibly could trigger that error message?
      • swlinn
        swlinn
        1344 Posts
        ACCEPTED ANSWER

        Re: Backend HTTPS connection error from MPGW

        ‏2013-01-14T17:54:09Z  in response to SystemAdmin
        I've seen this as more of a side effect log that the actual root cause. There may be some other root cause error log that indicates the failure, which of course then results in a response without a response header. Can you post your log records for this specific transaction?

        Regards,
        Steve
        • SystemAdmin
          SystemAdmin
          6772 Posts
          ACCEPTED ANSWER

          Re: Backend HTTPS connection error from MPGW

          ‏2013-01-14T18:50:22Z  in response to swlinn
          Thanks for looking into the issue. Here is the complete log. BTW I am running firmware 5.0.0.4

          20130110T053823Z mpgwinfo source-http(mpgwTest_FSH): tid(667649)http://1.2.3.4: Received HTTP/1.1 POST for verify/rest/events?access_token=a858973f-9c76-4fed-a3ba-699577107685 from 1.2.3.4
          20130110T053823Z mpgwdebug source-http(mpgwTest_FSH): tid(667649)http://1.2.3.4: HTTP Transaction # 1 on this TCP connection
          20130110T053823Z mpgwdebug source-http(mpgwTest_FSH): tid(667649)http://1.2.3.4: Found content length 0 HTTP input
          20130110T053823Z mpgwdebug Matching(All): tid(667649)http://1.2.3.4: Match: Received URL verify/rest/events?access_token=a858973f-9c76-4fed-a3ba-699577107685 matches rule '*'
          20130110T053823Z mpgwinfo stylepolicy(mpgwTest_policy): tid(667649)requesthttp://1.2.3.4: rule (mpgwTestRequestRule): selected via match 'All' from processing policy 'mpgwTest_policy'
          20130110T053823Z memory-reportdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: Request Started: memory used 71224
          20130110T053823Z memory-reportdebug mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: Processing finished: memory used 159728
          20130110T053823Z multistepinfo mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: rule (mpgwTestRequestRule): #1 convert-http: 'Converting INPUT with map (none) results stored in PIPE' completed OK.
          20130110T053823Z multistepdebug mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: Stylesheet URL to compile is 'local:///VerifyURLSetting.xsl'
          20130110T053823Z xsltdebug xmlmgr(OAUTH_XML_Manager): tid(667649)http://1.2.3.4: xslt Compilation Request: Checking cache for URL local:///VerifyURLSetting.xsl
          20130110T053823Z xsltdebug xmlmgr(OAUTH_XML_Manager): tid(667649)http://1.2.3.4: xslt Compilation Request: Found in cache (local:///VerifyURLSetting.xsl)
          20130110T053823Z mpgwinfo mpgw(mpgwTest): tid(667649)http://1.2.3.4: Destination URL is now https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z memory-reportdebug mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: Processing finished: memory used 296528
          20130110T053823Z multistepinfo mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: rule (mpgwTestRequestRule): #2 xform: 'Transforming PIPE with local:///VerifyURLSetting.xsl results stored in PIPE' completed OK.
          20130110T053823Z memory-reportdebug mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: Processing finished: memory used 297624
          20130110T053823Z multistepinfo mpgw(mpgwTest): tid(667649)requesthttp://1.2.3.4: rule (mpgwTestRequestRule): #3 results: 'generated from PIPE' completed OK.
          20130110T053823Z networkdebug xmlmgr(OAUTH_XML_Manager): tid(667649)http://1.2.3.4: Attempting TCP connect to abcd.efgh.gateway.com
          20130110T053823Z mpgwdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: HTTP Header-Retention:Header-Retention Policy for TE = ON. TE Header = (NULL), URL: https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: HTTP Header-Retention:Header-Retention Policy: Accept-Encoding = ON. Accept-Encoding Header = gzip,deflate, URL: https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: HTTP Header-Retention:Header-Retention Policy: Range = ON. Range Header = (NULL), URL: https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: HTTP Header-Retention:Header-Retention Policy:MQMD = ON. MQMD Header = (NULL), URL: https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: HTTP Header-Retention:Compression Policy: Off, URL: /amex/dpower/gateway?access_token=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: Outbound HTTP on new TCP session using HTTP/1.1 to https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z cryptoinfo valcred(mpgwTestSSLProfileValidationCredentials): tid(667649)http://1.2.3.4: certificate validation succeeded for '/CN=abcd.efgh.gateway.com' against 'mpgwTestSSLProfileValidationCredentials'
          20130110T053823Z memory-reportdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: Request Finished: memory used 159080
          20130110T053823Z mpgwinfo mpgw(mpgwTest): tid(667649)http://1.2.3.4: Using Backside Server: https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)http://1.2.3.4: The header (N/A) is wrong when connecting to URL ' https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          '.
          20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)http://1.2.3.4: Error occurred when connecting to URL ' https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          '
          20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)http://1.2.3.4: Backside header ('N/A') failed to parse due to: Failed to process response headers, URL: https://abcd.efgh.gateway.com:8443/abc?def=a858973f-9c76-4fed-a3ba-699577107685
          20130110T053823Z mpgwerror mpgw(mpgwTest): tid(667649)errorhttp://1.2.3.4: Failed to process response headers
          20130110T053823Z mpgwnotice stylepolicy(mpgwTest_policy): tid(667649)http://1.2.3.4: No error rule is matched.
          20130110T053823Z mpgwinfo mpgw(mpgwTest): tid(667649)errorhttp://1.2.3.4: No match from processing policy 'mpgwTest_policy' for code '0x01130011'
          20130110T053823Z memory-reportdebug mpgw(mpgwTest): tid(667649)http://1.2.3.4: Response Finished: memory used 93816
          • HermannSW
            HermannSW
            4238 Posts
            ACCEPTED ANSWER

            Re: Backend HTTPS connection error from MPGW

            ‏2013-01-15T13:11:12Z  in response to SystemAdmin
            "... The header (N/A) is wrong when ..."
            There is an error because you either get back an empty or wrong response from the backend.

            Please take a packet capture, open it in Wireshark, select "tcp.port==8443" and do a "Follow TCP stream".
            In case your connection to backend is secured by SSL, either turn it off for the one request,
            or configure the private key in Wireshark (see "RSA keys list" here) and then do "Follow SSL session".

             
            Hermann<myXsltBlog/> <myXsltTweets/>
            • SystemAdmin
              SystemAdmin
              6772 Posts
              ACCEPTED ANSWER

              Re: Backend HTTPS connection error from MPGW

              ‏2013-01-15T19:00:36Z  in response to HermannSW
              Thanks Hermann.

              I can't do either of these.

              Wireshark is a strict no no in our environment. It is against security policies.
              The backend support SSL only..we don't have a corresponding http port.

              Any other ideas?
              • kenhygh
                kenhygh
                1340 Posts
                ACCEPTED ANSWER

                Re: Backend HTTPS connection error from MPGW

                ‏2013-01-15T19:19:15Z  in response to SystemAdmin
                Be aware that what Hermann's asked you to do is not to install the part of Wireshark that does network sniffing, but the part that decodes and displays the packet capture from DataPower. That shouldn't run foul of your security policies.

                Ken
              • Trey
                Trey
                224 Posts
                ACCEPTED ANSWER

                Re: Backend HTTPS connection error from MPGW

                ‏2013-01-16T00:51:23Z  in response to SystemAdmin
                I would go ahead and see if you can work with the network admin or as Ken and Hermann noted just use the decode portion of wireshark. Once you have a good packet trace of a connection attempt we can see a little more, what is strange in your error is that we appear to get through the SSL handshake but as you noted are immediately shut down. This behavior is very strange and I tried to reproduce it but could not.

                If you can get a packet trace that would be key piece of data. Also I might suggest based on this thread that you consider a creating a pmr. I would be glad to help in any way that I can.

                Within the packet trace I would really want to see from the devices point of view how far the connection goes and IF we really do send any request payload toward the server.

                An option you may want to try.. again this would be to tinker with that backend only and feel it out, try using openssl s_client to connect and see how it behaves.
                openssl s_client -connect abcd.efgh.gateway.com:8443
                <if you get through the handshake type the following>
                GET /abc?def=a858973f-9c76-4fed-a3ba-699577107685 HTTP/1.1
                HOST= abcd.efgh.gateway.com:8443
                <hit enter twice>
                Take a look at the content you get from the server.

                Hope it helps.
                • SystemAdmin
                  SystemAdmin
                  6772 Posts
                  ACCEPTED ANSWER

                  Re: Backend HTTPS connection error from MPGW

                  ‏2013-01-16T10:31:24Z  in response to Trey
                  Thanks everyone. I am working with wireshark now. Meanwhile I opened up a PMR as well.
                  • P Anand
                    P Anand
                    1 Post
                    ACCEPTED ANSWER

                    Re: Backend HTTPS connection error from MPGW

                    ‏2014-03-18T09:30:46Z  in response to SystemAdmin

                     

                    Hi, We came across same issue recently while connecting to a backend over SSL. Non-persistent connections did not helped. Later issue was narrowed by disabling client cert validation on server end, which worked fine.

                    Backend admin mentioned it appears to be a configuration problem of Apache under Windows machine used for POC.

                    It was resolved by placing direct issuer under CA certificates path instead of intermediate, secondary and root 3 different files.

                    Did you got response from PMR on were there any way to update DP objects to get descriptive error log since it will avoid to go to backend if similar issue repeats.

                    -       thanks, Anand