Topic
  • 16 replies
  • Latest Post - ‏2013-11-01T16:20:58Z by JoeMorganNTST
JoeMorganNTST
JoeMorganNTST
427 Posts

Pinned topic Transform SOAP into PURE XML

‏2013-10-28T22:19:26Z |

This seems ridiculously simple, but I've been trying all day now and still cannot get it right.

This is what I have:

<soapenv:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <content xsi:type="xsd:string">
          <event>
              <test>Hello Joe</test>
              <test2>No way, Jose</test2>
          </event>
      </content>
   </soapenv:Body>
</soapenv:Envelope>

 

This is what I need after the transform:
 

<event>
   <test>Hello Joe</test>
   <test2>No way, Jose</test2>
</event>

 

  • kenhygh
    kenhygh
    2087 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-28T23:12:37Z  

    untested:

    <xsl:copy-of select="/*[local-name()='Envelope']/*[local-name()='Body']/*[local-name()='content']/*"/>

  • HermannSW
    HermannSW
    6205 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-29T03:48:06Z  

    Hi Joe,

    Ken's XPath is doing what you want:

    $ xpath++ "/*[local-name()='Envelope']/*[local-name()='Body']/*[local-name()='content']/*" Joe.xml

    -------------------------------------------------------------------------------
    <event xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
                  <test>Hello Joe</test>
                  <test2>No way, Jose</test2>
              </event>
    $


    You can verify easily by using xpath++ tool, see
    https://www.ibm.com/developerworks/community/blogs/HermannSW/tags/xpath%2B%2B
    http://stamm-wilbrandt.de/en/xsl-list/xpath++


    DataPower provides "Build XPath Expression from sample XML File" tool hidden eg. in "XPath Routing Map".

    You can open directly by "https://yourBox:yourWebGUIport/service/SelectXPath", see slide 3 of:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019118

    With your example clicking on "event" results in this longer, but more precise XPath:

    $ xpath++ "/*[namespace-uri()='http://schemas.xmlsoap.org/soap/envelope/' and local-name()='Envelope']/*[namespace-uri()='http://schemas.xmlsoap.org/soap/envelope/' and local-name()='Body']/content/event" Joe.xml

    -------------------------------------------------------------------------------
    <event xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
                  <test>Hello Joe</test>
                  <test2>No way, Jose</test2>
              </event>
    $


    Of course the XPath generated can be used as start for your own simplifications.

     


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>


     

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-29T19:45:48Z  
    • kenhygh
    • ‏2013-10-28T23:12:37Z

    untested:

    <xsl:copy-of select="/*[local-name()='Envelope']/*[local-name()='Body']/*[local-name()='content']/*"/>

    OK... My problem was the actual payload was encoded XML.. and when I copied their test message and put it into SoapUI, it "decoded" the request and was sending *actual* XML.

    When it is encoded, it's royally screwed up.  Straight XML, it's fine.

    But, strange as it might seem, when I run this now, everything works fine.  A SOAP message comes in, I do my stuff, send the XML to the server.  The server responds with simply "OK", which I ignore, and create a SOAP response back. 

    We are only receiving this:

    <?xm

    ... And that is it!  Can someone shed some light on this one?

     

  • HermannSW
    HermannSW
    6205 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-29T22:11:45Z  

    OK... My problem was the actual payload was encoded XML.. and when I copied their test message and put it into SoapUI, it "decoded" the request and was sending *actual* XML.

    When it is encoded, it's royally screwed up.  Straight XML, it's fine.

    But, strange as it might seem, when I run this now, everything works fine.  A SOAP message comes in, I do my stuff, send the XML to the server.  The server responds with simply "OK", which I ignore, and create a SOAP response back. 

    We are only receiving this:

    <?xm

    ... And that is it!  Can someone shed some light on this one?

     

    Hi Joe,

    it seems that the response contained an xml declaration and got truncated.

    Please do a packet capture to identify whether the backend does return this truncated XML, or whether DataPower does truncate.

    In case DataPower does, please attach some details (response from backend, firmware version, sample DataPower service) to investigate.

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
     

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T00:14:13Z  
    • HermannSW
    • ‏2013-10-29T22:11:45Z

    Hi Joe,

    it seems that the response contained an xml declaration and got truncated.

    Please do a packet capture to identify whether the backend does return this truncated XML, or whether DataPower does truncate.

    In case DataPower does, please attach some details (response from backend, firmware version, sample DataPower service) to investigate.

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
     

    It would have to be DataPower, somewhere.  The actual backend returns a plaintext "OK" that the DP processing rule ignores.  It then creates a SOAP response.

    The SOAP response is real simple, generated by this XSL:

    <xsl:stylesheet version="1.0"
        xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
        xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        
        <xsl:output method="xml" indent="yes" />
        
        <xsl:template match="/">
            <soapenv:Envelope
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
                <soapenv:Header/>
                <soapenv:Body>
                    <content xsi:type="xsd:string" xmlns="http://www.ntst.com/MHPKioskIntakeFormService">
                        OK
                    </content>
                </soapenv:Body>
            </soapenv:Envelope>
        </xsl:template>
    </xsl:stylesheet>

     

  • HermannSW
    HermannSW
    6205 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T01:05:59Z  

    It would have to be DataPower, somewhere.  The actual backend returns a plaintext "OK" that the DP processing rule ignores.  It then creates a SOAP response.

    The SOAP response is real simple, generated by this XSL:

    <xsl:stylesheet version="1.0"
        xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
        xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
        
        <xsl:output method="xml" indent="yes" />
        
        <xsl:template match="/">
            <soapenv:Envelope
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
                <soapenv:Header/>
                <soapenv:Body>
                    <content xsi:type="xsd:string" xmlns="http://www.ntst.com/MHPKioskIntakeFormService">
                        OK
                    </content>
                </soapenv:Body>
            </soapenv:Envelope>
        </xsl:template>
    </xsl:stylesheet>

     

    Hi Joe,

    how can you process a plaintext "OK" with Transform Action stylesheet?

    Can you attach a small recreate service?

     

    > We are only receiving this
    >

    <?xm

    > ... And that is it!  Can someone shed some light on this one?
    >

    Please do a packet capture on the frontside -- reading this it seems possible that your client truncates ...


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Updated on 2013-10-30T01:08:48Z at 2013-10-30T01:08:48Z by HermannSW
  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T01:09:56Z  
    • HermannSW
    • ‏2013-10-30T01:05:59Z

    Hi Joe,

    how can you process a plaintext "OK" with Transform Action stylesheet?

    Can you attach a small recreate service?

     

    > We are only receiving this
    >

    <?xm

    > ... And that is it!  Can someone shed some light on this one?
    >

    Please do a packet capture on the frontside -- reading this it seems possible that your client truncates ...


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    I've got the rule set for Non-XML processing, and the transform action with the above stylesheet doesn't take the INPUT, but takes NULL.  It spits out the SOAP, and PIPES it to the results action.

     

  • HermannSW
    HermannSW
    6205 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T17:49:47Z  

    I've got the rule set for Non-XML processing, and the transform action with the above stylesheet doesn't take the INPUT, but takes NULL.  It spits out the SOAP, and PIPES it to the results action.

     

    Hi Joe,

    I created MPGW Joe with response rule processing as you described.

    Request rule is pass-thru and goes against local loopback pass-thur XML FW JoeLB.

    This is the output I get (with 5.0.0.10 firmware, service export is attached):

    $ curl --data-binary "OK" http://dp1-l2:3101 ; echo
    <?xml version="1.0" encoding="UTF-8"?>

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soapenv:Header/>
    <soapenv:Body>
    <content xsi:type="xsd:string" xmlns="http://www.ntst.com/MHPKioskIntakeFormService">
                        OK
                    </content></soapenv:Body></soapenv:Envelope>OK
    $


    As you can see "OK" is appended to the SOAP response generated by your stylesheet reading from NULL.

    That is expected since you did not "consume" the Non-XML response rule input although you stated response type as Non-XML. Consumption can happen by convert-http action, Transform binary action, Cryptobin action ... . DataPower behaviour in case of Non-XML request or response type with input not being consumed is to just append the input to the generated output of the rule.

    Slide 15 of this WSTE webcast makes use of this feature by "prepend=" trick allowing a XS40 box to do Non-XML processing:
    http://www-01.ibm.com/support/docview.wss?uid=swg27022979


    So I cannot recreate your truncation and I am very interested in it.

    If you do have a (simplified) recreate, please either attach service export here, or create a PMR so that I can investigate.
    (I did fix some output truncation APARs in the past, but that was quite some time ago and should not happen if you are not on a backlevel firmware version)


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Attachments

    Updated on 2013-10-30T17:52:17Z at 2013-10-30T17:52:17Z by HermannSW
  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T19:54:03Z  
    • HermannSW
    • ‏2013-10-30T17:49:47Z

    Hi Joe,

    I created MPGW Joe with response rule processing as you described.

    Request rule is pass-thru and goes against local loopback pass-thur XML FW JoeLB.

    This is the output I get (with 5.0.0.10 firmware, service export is attached):

    $ curl --data-binary "OK" http://dp1-l2:3101 ; echo
    <?xml version="1.0" encoding="UTF-8"?>

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soapenv:Header/>
    <soapenv:Body>
    <content xsi:type="xsd:string" xmlns="http://www.ntst.com/MHPKioskIntakeFormService">
                        OK
                    </content></soapenv:Body></soapenv:Envelope>OK
    $


    As you can see "OK" is appended to the SOAP response generated by your stylesheet reading from NULL.

    That is expected since you did not "consume" the Non-XML response rule input although you stated response type as Non-XML. Consumption can happen by convert-http action, Transform binary action, Cryptobin action ... . DataPower behaviour in case of Non-XML request or response type with input not being consumed is to just append the input to the generated output of the rule.

    Slide 15 of this WSTE webcast makes use of this feature by "prepend=" trick allowing a XS40 box to do Non-XML processing:
    http://www-01.ibm.com/support/docview.wss?uid=swg27022979


    So I cannot recreate your truncation and I am very interested in it.

    If you do have a (simplified) recreate, please either attach service export here, or create a PMR so that I can investigate.
    (I did fix some output truncation APARs in the past, but that was quite some time ago and should not happen if you are not on a backlevel firmware version)


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Interesting.  The best I can tell you is that this is an XML FW, and I suspect it is what is stripping the payload.

    The actual transaction path is that a SOAP message comes through an MPG in an XB62 in the DMZ.  This validates a signature, removes the SOAP header, then passes what's left to an XML FW in an XI52 in our ST environment. 

    The XMLFW yanks out the XML body of a single content element and passes the RAW XML out to the back end server.  The Back-End server responds with a plaintext "OK".  As has been described, I just ignore it and produce the static SOAP output.

    In my case, as least from the point of view of the probe, it isn't appending an OK. 

    If I set the MPG in the XB62 to anything other than Passthrough, the message errors.  So, I can only suspect it is happening in the XML FW (export attached). 

    Attachments

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T20:10:23Z  

    Interesting.  The best I can tell you is that this is an XML FW, and I suspect it is what is stripping the payload.

    The actual transaction path is that a SOAP message comes through an MPG in an XB62 in the DMZ.  This validates a signature, removes the SOAP header, then passes what's left to an XML FW in an XI52 in our ST environment. 

    The XMLFW yanks out the XML body of a single content element and passes the RAW XML out to the back end server.  The Back-End server responds with a plaintext "OK".  As has been described, I just ignore it and produce the static SOAP output.

    In my case, as least from the point of view of the probe, it isn't appending an OK. 

    If I set the MPG in the XB62 to anything other than Passthrough, the message errors.  So, I can only suspect it is happening in the XML FW (export attached). 

    To follow up, I think I know what was happening.  If I add omit-xml-declaration="yes" to the stylesheet, it no longer truncates the so much of the response, and I'm witnessing the appending you are talking about.

    In the new test, I didn't get back a plain text OK, but some kind of HTML response.  That is truncated to an extent.  It includes a Java stacktrace, and that is cut off about 1/2 of the way through.

     

     

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T20:34:47Z  

    To follow up, I think I know what was happening.  If I add omit-xml-declaration="yes" to the stylesheet, it no longer truncates the so much of the response, and I'm witnessing the appending you are talking about.

    In the new test, I didn't get back a plain text OK, but some kind of HTML response.  That is truncated to an extent.  It includes a Java stacktrace, and that is cut off about 1/2 of the way through.

     

     

    OK... maybe I was wrong.  My payload was going out wrong... causing the error, that was being appended and still truncated.  I fixed the payload, still omitting the XML declaration, but now it is getting truncated with just a LF and then "<so"

    So... it really looks like it is truncating to exactly 4 characters.

  • kenhygh
    kenhygh
    2087 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-30T22:30:11Z  

    OK... maybe I was wrong.  My payload was going out wrong... causing the error, that was being appended and still truncated.  I fixed the payload, still omitting the XML declaration, but now it is getting truncated with just a LF and then "<so"

    So... it really looks like it is truncating to exactly 4 characters.

    highly unlikely some bug in DP is doing this. what's probe/packet capture showing?

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-31T14:50:44Z  
    • kenhygh
    • ‏2013-10-30T22:30:11Z

    highly unlikely some bug in DP is doing this. what's probe/packet capture showing?

    The probe shows the complete message going back.  The MPG in the DMZ is set to passthrough, and right now, I cannot change it or I break a whole bunch of stuff being tested.

    I need to do the packet capture.  I suspect I need to capture on BOTH appliances?

    I cannot do XML File capture.  If I turn that on for only a few seconds the appliance reboots.  That feature needs to be improved to write to the raid array or something so it is useful and not so unstable. 

  • kenhygh
    kenhygh
    2087 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-10-31T21:46:36Z  

    The probe shows the complete message going back.  The MPG in the DMZ is set to passthrough, and right now, I cannot change it or I break a whole bunch of stuff being tested.

    I need to do the packet capture.  I suspect I need to capture on BOTH appliances?

    I cannot do XML File capture.  If I turn that on for only a few seconds the appliance reboots.  That feature needs to be improved to write to the raid array or something so it is useful and not so unstable. 

    Joe,

    Can you test without going through the MPG in the DMZ? is this happening only in Prod? do you have a unit test that demonstrates the failure?

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-11-01T14:17:30Z  
    • kenhygh
    • ‏2013-10-31T21:46:36Z

    Joe,

    Can you test without going through the MPG in the DMZ? is this happening only in Prod? do you have a unit test that demonstrates the failure?

    Not in Prod, fortunately, but there are half of a dozen development projects in the works, and we have a strange environment where we *share* appliances for development/QA, and in other cases, QA/Production.  RIght now, we have a pretty significant QA going on, and can't have the box fail.  If I'm going to do captures, I'll have to do it over the weekend.

    You did give me an idea.. maybe I should migrate the XMLFW to an MPG and see if there is any difference.

     

  • JoeMorganNTST
    JoeMorganNTST
    427 Posts

    Re: Transform SOAP into PURE XML

    ‏2013-11-01T16:20:58Z  
    • kenhygh
    • ‏2013-10-31T21:46:36Z

    Joe,

    Can you test without going through the MPG in the DMZ? is this happening only in Prod? do you have a unit test that demonstrates the failure?

    OK... I know the problem, not exactly sure how to *accurately* correct it.

    It is respecting the Content-Length header.  When the server returns "OK", that is appended with a CR/LF, thus giving me exactly 4 characters.  If I change the Content-Length header, it fixes it.  Kind of.

    When the server returns a "successful" response, I get back a content length of 4.  If there is an error, the content length is whatever.

    I guess I'm going to have to write something that dynamically expands the content length based upon what I get back.

    Either way... it is truncating at the XML Firewall because of the Content-Length header.