Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
28 replies Latest Post - ‏2014-02-18T20:56:50Z by Aprithvi
Aprithvi
Aprithvi
52 Posts
ACCEPTED ANSWER

Pinned topic Url-open illegal character > at offset

‏2013-12-01T13:11:24Z |

HI I use Url-open in a stylesheet to connect to a webservice. Sporadically I get an http response code 500 with the message illegal character > at offset 49. Im using response =responsecode in my xsl. What could be the reason for this? I get the same error for multiple webservices.

  • HermannSW
    HermannSW
    4488 Posts
    ACCEPTED ANSWER

    Re: Url-open illegal character > at offset

    ‏2013-12-01T14:38:55Z  in response to Aprithvi

    Hi Anju,

    from your profile's Recent Updates tab I saw that your last question was related to FTP.

    Infocenter gives all the responses for the different protocols in case of  response=responsecode:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xi.doc%2Fextensionfunctions35.htm

    In case you did open a FTP URL only the unsuccessful case may result in a problem:

    All failed FTP transactions return the following element, when you set the response attribute to responsecode:

    
    <url-open>
      <response> ... returned data ... </response>
    </url-open>
    


    The only problem I can think of matching your error message is that "returned data" is text containing a '>' which makes above response Non-XML.

    You need to find out what exactly gets returned by your dp:url-open.

    If in production, please do not enable Probe -- you need to take packet captures there.

    If in dev/test environment, you may enable Probe and hope to capture a failing transaction.

    All depends on the frequency of the failure, and whether you know how to possible enforce or make a failure more likely.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>
     

    • Aprithvi
      Aprithvi
      52 Posts
      ACCEPTED ANSWER

      Re: Url-open illegal character > at offset

      ‏2013-12-02T15:18:48Z  in response to HermannSW

      Thank you Hermann. The url-open problem here is with an HTTP url. We have response =responsecode in the xsl. In the probe I can only see this error  "illegal character '>' at offset 49 of ". We are making a soap over http call to these services and multiple services return the same error. On the services side we dont see any error. How do I see the response being returned to datapower? 

      Full response

      <url-open><responsecode>500</responsecode><errorcode>196609</errorcode><errorstring>illegal character '>' at offset 49 of http://10.87.16.19:3000/abcservice</errorstring><content-type>text/html; charset=iso-8859-1</content-type><headers><header name="Date">Mon, 02 Dec 2013 06:03:36 GMT</header><header name="Server">IBM_HTTP_Server</header><header name="Content-Type">text/html; charset=iso-8859-1</header></headers></url-open>

       

      • HermannSW
        HermannSW
        4488 Posts
        ACCEPTED ANSWER

        Re: Url-open illegal character > at offset

        ‏2013-12-02T15:49:52Z  in response to Aprithvi

        Hi,

        what does your dp:url-open look like?

        > How do I see the response being returned to datapower?
        >

        Please take a packet capture to see response.


        Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

         


         

        • Aprithvi
          Aprithvi
          52 Posts
          ACCEPTED ANSWER

          Re: Url-open illegal character > at offset

          ‏2013-12-02T16:08:54Z  in response to HermannSW

          You mean Export Capture? Where do I take the packet capture?

          • Aprithvi
            Aprithvi
            52 Posts
            ACCEPTED ANSWER

            Re: Url-open illegal character > at offset

            ‏2013-12-02T16:12:08Z  in response to Aprithvi

            Also this behavior does not happen all the time. When I take the same request and run it again through datapower the response is fine.

            Url open looks like this

            <dp:url-open target="{$endpointURL}" response="responsecode" http-headers="$headerValues" http-method="post">
            <xsl:copy-of select="$parsed" />
            </dp:url-open>

             

            • HermannSW
              HermannSW
              4488 Posts
              ACCEPTED ANSWER

              Re: Url-open illegal character > at offset

              ‏2013-12-02T16:58:04Z  in response to Aprithvi

              In WebGUI "Troubleshooting", "Main" tab, "Packet Capture" section.

              You may analyze the downloaded "*.pcap" file eg. by Wireshark tool.


              Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

               

              Updated on 2013-12-02T16:58:48Z at 2013-12-02T16:58:48Z by HermannSW
            • swlinn
              swlinn
              1346 Posts
              ACCEPTED ANSWER

              Re: Url-open illegal character > at offset

              ‏2013-12-02T19:41:22Z  in response to Aprithvi

              I've seen instances where the backend server container could have thread issues that prevents it periodically from servicing the request, so it returns a HTML response which the parser most likely will not like.  As Hermann said, get a packet capture (from troubleshooting in the default domain). Specify a filter of "Host 1.2.3.4" where 1.2.3.4 is the IP of your backend server.  That way the capture will only have packets flowing to/from your backend and not other servers.  You'll see what's on the wire in the failing case, specifically if the response is HTML as I suspect.  Since the issue is intermittent, you should setup the capture to be continuous, and once you see the issue, you can stop the packet capture.  For extra credit, you can set up a trigger that will catch your parsing error code and will issue a CLI command in this case to stop the packet capture for you :-) but otherwise, just stop the capture from the GUI when you see the problem.

              Regards,
              Steve

              • Aprithvi
                Aprithvi
                52 Posts
                ACCEPTED ANSWER

                Re: Url-open illegal character > at offset

                ‏2013-12-02T21:57:56Z  in response to swlinn

                Yes we did see an HTML response when I triggered from Soap UI. but the url-open response code translates it to 

                <errorstring>illegal character '>' at offset 49 ofhttp://10.87.16.19:3000/abcservice</errorstring>

                How do I get the exact html response back in datapower?

                • swlinn
                  swlinn
                  1346 Posts
                  ACCEPTED ANSWER

                  Re: Url-open illegal character > at offset

                  ‏2013-12-02T22:19:49Z  in response to Aprithvi

                  Hermann has plenty of examples in this forum and his blog on processing non-XML data when using a url-open, but the fastest way is to get the packet capture and look at the results in wireshark.

                  Regards,

                  Steve

                  • Aprithvi
                    Aprithvi
                    52 Posts
                    ACCEPTED ANSWER

                    Re: Url-open illegal character > at offset

                    ‏2013-12-09T20:36:04Z  in response to swlinn

                    Swlinn,

                    we made a change in the xml firewall from requesttype XML to non xml. with this change we get the actual response back with a response code of 200. When it is XML we get 500. So it means the url-open was able to contact the webservice but not able to parse the response. How do we handle this?

                    • swlinn
                      swlinn
                      1346 Posts
                      ACCEPTED ANSWER

                      Re: Url-open illegal character > at offset

                      ‏2013-12-09T21:24:01Z  in response to Aprithvi

                      Hi Aprithvi,

                      I'm confused.  You're saying that by changing the service request type from XML to Non-XML that the response of the url-open is properly parsed, eg, that the backend is sending a XML response only when the service request type is Non-XML?  Regardless of what the service request type is, the url-open response parsing should be based on the response attribute in the url-open extension function. 

                      I thought your issue was you were receiving a non-XML response from the url-open target server periodically.  You were able to see that response in Soap-UI.  What type of error does this response indicate?  Ultimately I would think it would indicate a problem with the target server that would need to be addressed, for example, lack of container threads.   If you need to see this HTML response in DataPower as the result of a url-open, I would believe that you should use the url-open response="binaryNode" attribute.  Hermann has put a number of posts in this forum on processing non-XML data and binaryNode in xslt.   https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014572518#77777777-0000-0000-0000-000014626866 looks like a good reference post in this forum, and it has links to his blog and some webcasts he's had on Non-XML data processing which may be helpful.

                      Regards,

                      Steve

                      • Aprithvi
                        Aprithvi
                        52 Posts
                        ACCEPTED ANSWER

                        Re: Url-open illegal character > at offset

                        ‏2013-12-10T13:15:30Z  in response to swlinn

                        yes that is correct. With response type as responsecode we get a 200 only when the request type in xml firewall is Non-xml. and the response is a valid XML data. we really need to capture what url-open response is in the probe. the illegal character is after url-open parses the response? how do we tell url-open not to parse response and see wht the actual service is sending back?

                        • HermannSW
                          HermannSW
                          4488 Posts
                          ACCEPTED ANSWER

                          Re: Url-open illegal character > at offset

                          ‏2013-12-10T13:35:17Z  in response to Aprithvi

                          Slide 13 shows that  response="binaryNode"  is needed.

                          Slide 12 maginifying-glass.xsl shows how to display binary data hexadecimally encoded (only safe way for binary).

                          http://www-01.ibm.com/support/docview.wss?uid=swg27022977

                          This is the corresponding advanced webcast;
                          http://www-01.ibm.com/support/docview.wss?uid=swg27022979

                          Please hear into the audio recording for details, samples are attached.


                          Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

                           

                          • Aprithvi
                            Aprithvi
                            52 Posts
                            ACCEPTED ANSWER

                            Re: Url-open illegal character > at offset

                            ‏2013-12-22T17:35:27Z  in response to HermannSW

                            Hermann,

                             

                            Im using url-open with response as binary node

                            <dp:url-open target="{$endpointURL}" response="binaryNode" content-type="{$paramContentType}" http-headers="$headerValues" http-method="post">
                            <xsl:copy-of select="$parsed"/>
                            </dp:url-open>

                             

                            the response i get is

                            <result><binary>**binary node**</binary>

                             

                            The data in the binary node could be xml or non xml. How do I retrive that data

                            I did this

                            <xsl:variable name ="binary">
                            <xsl:copy-of select="$serviceResponse/result/binary/child::node()"/>
                             
                            </xsl:variable>
                            <xsl:variable name="base64">
                            <xsl:copy-of select="dp:binary-decode($binary)"/>

                            I dont get anything in binary at all. how do I retrieve the real data from **binary node**?

                            • HermannSW
                              HermannSW
                              4488 Posts
                              ACCEPTED ANSWER

                              Re: Url-open illegal character > at offset

                              ‏2013-12-22T17:48:12Z  in response to Aprithvi

                              Hi,

                              you cannot use "dp:binary-decode()" in a stylesheet to get the "value" of a base64 string.

                              In case the content of binary node is UTF-8 encoded, you can use "dp:decode($binary, 'utf-8')".
                              But it does UTF-8 validation and error out in case of Non-UTF-8 encoded data.

                              For other encodings you may use techniques described on slides 4-8 of this WSTE webcast
                              (hear into audio recording for details, samples files are attached, too):
                              http://www-01.ibm.com/support/docview.wss?uid=swg27022979

                              You cannot process unencoded (base64, hex, ...) general binary data in XSLT.


                              Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>
                               

                              Updated on 2013-12-22T17:49:26Z at 2013-12-22T17:49:26Z by HermannSW
                              • Aprithvi
                                Aprithvi
                                52 Posts
                                ACCEPTED ANSWER

                                Re: Url-open illegal character > at offset

                                ‏2014-02-18T16:23:18Z  in response to HermannSW

                                Hello Herman,

                                This issue is still not resolved. We are trying to process xml and non xml in the same service using dp:url-open in a stylesheet. We have a routing stylesheet that does a dp-transform on another stylesheet that is calling a service through url-open. 

                                When I use output- mapping to process binary it throws me out

                                Use of dp:output-mapping (binary output) in 'local:///output/pcg-esb/xsl/esb_soapcall.xsl'disallowed in dp:transform

                                This is how we are doing 

                                xsl1 calls xsl2 via dp-transform

                                xsl2 calls a service through url-open and the response could be xml or non xml.

                                How do I process this situation in xsl2? I could use any help in this regards. Thank you for looking.

                                • HermannSW
                                  HermannSW
                                  4488 Posts
                                  ACCEPTED ANSWER

                                  Re: Url-open illegal character > at offset

                                  ‏2014-02-18T16:56:26Z  in response to Aprithvi

                                  dp:transform() is for XSLT transformation of XML and does not support <dp:input-mapping> or <dp:output-mapping> for dealing with binary data.

                                  If you need <dp:output-mapping> and your description sounds like that, then you need to use a chained service with Transform Binary action.


                                  Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/> <GraphvizFiddle/> <xqib/>

                                  Updated on 2014-02-18T16:56:58Z at 2014-02-18T16:56:58Z by HermannSW
                                  • Aprithvi
                                    Aprithvi
                                    52 Posts
                                    ACCEPTED ANSWER

                                    Re: Url-open illegal character > at offset

                                    ‏2014-02-18T18:10:50Z  in response to HermannSW

                                    in our xsl2 we are calling a webserver which internally calls an app server. When appserver does not return a response the webserver throws an internal server error. 500

                                    in xsl2 this internal server error translates to illegal character >

                                    We want to capture the actual html that comes from the webserver

                                    For that we tried the url-open with response ='binaryNode'. do we need the   <dp:output-mapping> in that case>

                                    when response code is 200 we get the **BINARY NODE** in the response

                                    result/binary. But otherwise it is empty. Could you please explain more on the chained service? for this setup since we are using route action which has xsl1.

                                    • HermannSW
                                      HermannSW
                                      4488 Posts
                                      ACCEPTED ANSWER

                                      Re: Url-open illegal character > at offset

                                      ‏2014-02-18T19:07:01Z  in response to Aprithvi

                                      > For that we tried the url-open with response ='binaryNode'. do we need the   <dp:output-mapping> in that case?
                                      >
                                      no.

                                      > when response code is 200 we get the **BINARY NODE** in the responseresult/binary. But otherwise it is empty.
                                      >
                                      Use "dp:binary-encode($resp/result/binary)" to access the content of the binary node. See the examples in the Webcasts I referred
                                      to, eg. slide 19 of 2nd webcast.


                                      Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/> <GraphvizFiddle/> <xqib/>
                                       

                                      • Aprithvi
                                        Aprithvi
                                        52 Posts
                                        ACCEPTED ANSWER

                                        Re: Url-open illegal character > at offset

                                        ‏2014-02-18T19:17:17Z  in response to HermannSW

                                        I am getting the content as empty. Please see below. all xsl message are coming empty. If I change the response="responsecode" we get the xml back.

                                        <dp:url-open target="{$endpointURL}" response="binaryNode" http-headers="$headerValues" http-method="post">
                                        <xsl:copy-of select="$parsed"/>
                                        </dp:url-open>
                                        <xsl:variable name="encodedResult" select="dp:binary-encode($serviceResponse/result/binary)"/>
                                         
                                        <xsl:variable name="rawResult" select="dp:decode($encodedResult,'base-64')"/>
                                        <xsl:variable name="xml"> <xsl:copy-of select="dp:parse($rawResult,'base-64')"/> </xsl:variable> 
                                        <xsl:message dp:priority="debug"> 
                                                        *** Raw data from url-open is: <xsl:value-of select="$rawResult"/> 
                                            </xsl:message> 
                                            <xsl:message dp:priority="debug"> 
                                                        *** encoded data from url-open is: <xsl:copy-of select="$encodedResult"/> 
                                            </xsl:message> 
                                            <xsl:message dp:priority="debug"> 
                                                        *** XML data from url-open is: <xsl:value-of select="$xml"/> 
                                            </xsl:message>     <xsl:message dp:priority="debug"> 
                                                        *** Service responsedata from url-open is: <xsl:copy-of select="$serviceResponse"/> 
                                            </xsl:message> 
                                            
                                        • HermannSW
                                          HermannSW
                                          4488 Posts
                                          ACCEPTED ANSWER

                                          Re: Url-open illegal character > at offset

                                          ‏2014-02-18T19:27:54Z  in response to Aprithvi

                                          Do you get message "*** encoded data from url-open is:" in the log?
                                          Or does execution fail before.

                                          dp:decode(_,'base-64') does UTF-8 validation and fails in case binary data does not represent a UTF-8 encoded string.

                                          Just output $encodedResult in that case and you will see a (possibly empty) base64 string. "binaryNode" response is definitely able to return ANY binary data response, so html cannot be a problem. If $encodedResult is empty string, then no data was returned from backend. You can prove that by taking a packet capture and doing a "Follow TCP Stream" in Wireshark for backend connection.


                                          Hermann.

                                          • Aprithvi
                                            Aprithvi
                                            52 Posts
                                            ACCEPTED ANSWER

                                            Re: Url-open illegal character > at offset

                                            ‏2014-02-18T19:39:59Z  in response to HermannSW

                                            yes I get the message

                                             "*** encoded data from url-open is:" in the log. It is empty.

                                            wsgw (ESB-PROXY)*** encoded data from url-open is: 

                                            And I know for sure the baceend is returning data because when I change to response=responsecode the xml content is there.

                                            Im totally stuck on this. All I need is to process xml and html in the same service.

                                            Using response=binarynode and the responsecode is 200 I get the below

                                            <result><binary>***BINARY NODE***</binary><responsecode>200</responsecode><headers><header name="Date">Tue, 18 Feb 2014 19:30:52 GMT</header><header name="Server">IBM_HTTP_Server</header><header name="SOAPAction">""</header><header name="Accept">text/xml</header><header name="Content-Type">text/xml</header><header name="Content-Language">en-US</header></headers></result>

                                            if the responsecode is not 200 I dont see anything in the probe and the error says Remote error on URL.

                                            • HermannSW
                                              HermannSW
                                              4488 Posts
                                              ACCEPTED ANSWER

                                              Re: Url-open illegal character > at offset

                                              ‏2014-02-18T19:54:09Z  in response to Aprithvi

                                              As I said, please take a packet capture and you will see that nothing gets returned if $encodedResponse is empty in case of HTTP 200 return.

                                              If the packet capture will prove that really nothing is returned, investigate your backend.

                                              Otherwise please create a PMR and Level 2 support will collect the needed artefacts for infouse recreation from you.


                                              Hermann.

                                              Updated on 2014-02-18T19:54:37Z at 2014-02-18T19:54:37Z by HermannSW
                                              • Aprithvi
                                                Aprithvi
                                                52 Posts
                                                ACCEPTED ANSWER

                                                Re: Url-open illegal character > at offset

                                                ‏2014-02-18T20:03:53Z  in response to HermannSW

                                                Thank you Herman for being so patient and answering. Please bare with me one more time. The only thing i change in the xsl is response=BinaryNode to response=responsecode. With that I get xml back. How is that possible

                                                • HermannSW
                                                  HermannSW
                                                  4488 Posts
                                                  ACCEPTED ANSWER

                                                  Re: Url-open illegal character > at offset

                                                  ‏2014-02-18T20:09:08Z  in response to Aprithvi

                                                  I have no idea, that is why I asked you to take packet capture.

                                                  Only packet capture will tell you the truth and only the truth.

                                                  If packet capture shows response data with binaryNode its PMR time, otherwise your backend needs investigation.

                                                  Hermann.

                                                  • Aprithvi
                                                    Aprithvi
                                                    52 Posts
                                                    ACCEPTED ANSWER

                                                    Re: Url-open illegal character > at offset

                                                    ‏2014-02-18T20:37:25Z  in response to HermannSW

                                                    ok but how about an HTTP 500 return? can we capture this with binaryNode?

                                                    when it is other than 200 I do not see any response in probe. Just the remote error on URL

                                                    • HermannSW
                                                      HermannSW
                                                      4488 Posts
                                                      ACCEPTED ANSWER

                                                      Re: Url-open illegal character > at offset

                                                      ‏2014-02-18T20:42:12Z  in response to Aprithvi

                                                      You have the config at hand -- do not trust Probe here, pelase take a packet capture.

                                                      It will tell us all.

                                                      • Aprithvi
                                                        Aprithvi
                                                        52 Posts
                                                        ACCEPTED ANSWER

                                                        Re: Url-open illegal character > at offset

                                                        ‏2014-02-18T20:56:50Z  in response to HermannSW

                                                        so should I be getting this in case response is not 200? and the ***BINARY NODE*** will contain html?

                                                        <result><binary>***BINARY NODE***</binary><responsecode>500</responsecode><headers><header name="Date">Tue, 18 Feb 2014 19:30:52 GMT</header><header name="Server">IBM_HTTP_Server</header><header name="SOAPAction">""</header><header name="Accept">text/xml</header><header name="Content-Type">text/html</header><header name="Content-Language">en-US</header></headers></result>