Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
28 replies Latest Post - ‏2011-08-29T06:20:56Z by charanchowdary
charanchowdary
charanchowdary
40 Posts
ACCEPTED ANSWER

Pinned topic Conversion of XML file to a comma separated ASCII file

‏2011-07-27T13:59:02Z |
Any pointers on converting XML file to a comma separated ASCII file? Can it be done using XSLT?

Thanks in advance...
...
RK
Updated on 2011-08-29T06:20:56Z at 2011-08-29T06:20:56Z by charanchowdary
  • JBrennan
    JBrennan
    77 Posts
    ACCEPTED ANSWER

    Re: Conversion of XML file to a comma separated ASCII file

    ‏2011-07-27T14:34:18Z  in response to charanchowdary
    Yes, this can be done is XSLT. Simply loop through the elements you want and append to one string with a comma between them. Since XSL variables are immutable you must think about recursion here.

    When you have the string build, simply use the xsl:value-of to output it. Be sure you have the <xsl:output method="text"/> at the beginning of the stylesheet.
  • HermannSW
    HermannSW
    4488 Posts
    ACCEPTED ANSWER

    Re: Conversion of XML file to a comma separated ASCII file

    ‏2011-08-02T00:35:41Z  in response to charanchowdary
    Hi,

    > Any pointers on converting XML file to a comma separated ASCII file? Can it be done using XSLT?
    >
    you got the answer for XML2CSV from Jim.

    I looked up what I have and found a solution I once did for the opposite direction CSV2XML.
    The beauty of this solution is that it is a single stylesheet version based on FFD file shipped with the product.

    Sample input file "support.txt":
    
    command \ model,XA35,XS40,XI50,XB60,XM70 convert-http,yes,yes,yes,yes,yes dp:input-mapping,no,no,yes,yes,yes dp:output-mapping,no,no,yes,yes,yes
    


    Beautified (by "tidy -q -xml") XML output:
    
    <table> <row> <col>command \ model</col> <col>XA35</col> <col>XS40</col> <col>XI50</col> <col>XB60</col> <col>XM70</col> </row> <row> <col>convert-http</col> <col>yes</col> <col>yes</col> <col>yes</col> <col>yes</col> <col>yes</col> </row> <row> <col>dp:input-mapping</col> <col>no</col> <col>no</col> <col>yes</col> <col>yes</col> <col>yes</col> </row> <row> <col>dp:output-mapping</col> <col>no</col> <col>no</col> <col>yes</col> <col>yes</col> <col>yes</col> </row> </table>
    


    And this is the binary transform stylesheet "csv2xmlbin.xsl":
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" xmlns:str=
    "http://exslt.org/strings" extension-element-prefixes=
    "dp str" > <dp:input-mapping  href=
    "store:///pkcs7-convert-input.ffd" type=
    "ffd"/>   <xsl:output omit-xml-declaration=
    "yes" />   <xsl:variable name=
    "rowsep" select=
    "'&#10;'"/> <xsl:variable name=
    "colsep" select=
    "','"/>   <xsl:template match=
    "/"> <xsl:variable name=
    "str" select=
    "dp:decode(dp:binary-encode(/object/message/node()),'base-64')"/>   <table> <xsl:for-each select=
    "str:split($str,$rowsep)"> <xsl:
    
    if test=
    "position()!=last()"> <row> <xsl:for-each select=
    "str:split(.,$colsep)"> <col><xsl:value-of select=
    "."/></col> </xsl:for-each> </row> </xsl:if> </xsl:for-each> </table> </xsl:template> </xsl:stylesheet>
    

    Hermann<myXsltBlog/>
    • charanchowdary
      charanchowdary
      40 Posts
      ACCEPTED ANSWER

      Re: Conversion of XML file to a comma separated ASCII file

      ‏2011-08-02T03:56:05Z  in response to HermannSW
      Thank you Hermann for your response.

      We are actually looking for a solution to OUTPUT the below comma separated file for the INPUT xml file and vice versa. (comma separated ASCII value file to xml file).

      Appreciate any help in this regard.

      INPUT:

      <DST xml:lang="en-US">
      <jobName version="1.0">Authenticate</jobName>
      <trace>0</trace>
      <readable>N</readable>
      </DST>

      OUTPUT:

      60, 68, 83, 84, 32, 120, 109, 108, 58, 108, 97, 110, 103, 61, 34, 101, 110, 45, 85, 83, 34, 62, 13, 10, 32, 32, 32, 32, 60, 106, 111, 98, 78, 97, 109, 101, 32, 118, 101, 114, 115, 105, 111, 110, 61, 34, 49, 46, 48, 34, 62, 65, 117, 116, 104, 101, 110, 116, 105, 99, 97, 116, 101, 60, 47, 106, 111, 98, 78, 97, 109, 101, 62, 13, 10, 32, 32, 32, 32, 60, 116, 114, 97, 99, 101, 62, 48, 60, 47, 116, 114, 97, 99, 101, 62, 13, 10, 32, 32, 32, 32, 60, 114, 101, 97, 100, 97, 98, 108, 101, 62, 78, 60, 47, 114, 101, 97, 100, 97, 98, 108, 101, 62, 13, 10, 60, 47, 68, 83, 84, 62, 32, 32, 32, 32
      • JBrennan
        JBrennan
        77 Posts
        ACCEPTED ANSWER

        Re: Conversion of XML file to a comma separated ASCII file

        ‏2011-08-02T12:58:08Z  in response to charanchowdary
        You can use what I posted to go from XML to CSV. Then Hermann gave you the reverse. TO go from CSV to XML you can also use the tokenize extenssion function.
      • HermannSW
        HermannSW
        4488 Posts
        ACCEPTED ANSWER

        Re: Conversion of XML file to a comma separated ASCII file

        ‏2011-08-02T13:27:21Z  in response to charanchowdary
        It seems that Jim and I misunderstood your request.
        So basically you want to generate a sequence of ascii-codes of your XML file.

        This can be done by serializing the XML, and then do some fancy stuff ;-)

        I really like the 2nd line (which prevents trailing comma) in the central "concat", Oliver Becker's method of conditional selection:
        http://www.xml.org/xml/xslt_efficient_programming_techniques.pdf#page=5
        (try to find out the effect of "1 div ..." yourself if you do not already know)

        The actual conversion to character codes is done by base64 encoding the serialized XML and then recursively determine the ascii codes.

        
        $ cat asc.xml <DST xml:lang=
        "en-US"> <jobName version=
        "1.0">Authenticate</jobName> <trace>0</trace> <readable>N</readable> </DST> $ $ coproc2 asc.xsl asc.xml http:
        //cosmopolitan.boeblingen.de.ibm.com:2223; echo 60,68,83,84,32,120,109,108,58,108,97,110,103,61,34,101,110,45,85,83,34,62,10,60,106,111,98,78,97,109,101,32,118,101,114,115,105,111,110,61,34,49,46,48,34,62,65,117,116,104,101,110,116,105,99,97,116,101,60,47,106,111,98,78,97,109,101,62,10,60,116,114,97,99,101,62,48,60,47,116,114,97,99,101,62,10,60,114,101,97,100,97,98,108,101,62,78,60,47,114,101,97,100,97,98,108,101,62,10,60,47,68,83,84,62 $ $ cat asc.xsl <xsl:stylesheet version=
        "1.0" xmlns:xsl=
        "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
        "http://www.datapower.com/extensions" xmlns:func=
        "http://exslt.org/functions" extension-element-prefixes=
        "dp" > <xsl:output omit-xml-declaration=
        "yes" /> <func:function name=
        "func:b64-to-dec"> <xsl:param name=
        "str"/>   <xsl:choose> <xsl:when test=
        "not($str)"/>   <xsl:otherwise> <func:result select=
        " concat( dp:radix-convert(dp:substring-base64($str, 1, 1), 64, 10), substring(
        ',', 1 div (substring($str, 3) != 
        '==')), func:b64-to-dec(dp:substring-base64($str, 2)) )
        "/> </xsl:otherwise> </xsl:choose> </func:function>   <xsl:template match=
        "/"> <xsl:variable name=
        "str"> <dp:serialize omit-xml-decl=
        "yes" select=
        "."/> </xsl:variable>   <xsl:value-of select=
        "func:b64-to-dec(dp:encode($str,'base-64'))"/> </xsl:template> </xsl:stylesheet> $
        


        Hermann<myXsltBlog/>
        • JBrennan
          JBrennan
          77 Posts
          ACCEPTED ANSWER

          Re: Conversion of XML file to a comma separated ASCII file

          ‏2011-08-02T13:57:29Z  in response to HermannSW
          ah yea, I misunderstood it completely. Very slick Hermann!
        • Liv2luv
          Liv2luv
          573 Posts
          ACCEPTED ANSWER

          Re: Conversion of XML file to a comma separated ASCII file

          ‏2011-08-02T14:22:20Z  in response to HermannSW
          Hi Hermann,

          I was looking at it a little differently, not as good view point as yours :)

          The xml message will be converted into a hex stream and passed to this XSL (see below) which outputs ASCII codes for corresponding hex character (read from a mapping xml document)

          
          <?xml version=
          "1.0" encoding=
          "UTF-8"?> <xsl:stylesheet xmlns:xsl=
          "http://www.w3.org/1999/XSL/Transform" version=
          "1.0"> <xsl:output method=
          "xml"/>   <xsl:template match=
          "root/*/text()"> <xsl:
          
          if test=
          "string-length(.) mod 2 = 0">   <xsl:variable name=
          "csvString"> <xsl:call-template name=
          "csvGen"> <xsl:with-param name=
          "index" select=
          "1"/> <xsl:with-param name=
          "text" select=
          "."/> <xsl:with-param name=
          "strLength" select=
          "string-length(.)"/> </xsl:call-template> </xsl:variable> <out> <xsl:value-of select=
          "concat('[',$csvString,']')"/> </out> </xsl:if> </xsl:template> <xsl:template name=
          "csvGen"> <xsl:param name=
          "index"/> <xsl:param name=
          "text"/> <xsl:param name=
          "strLength"/> <xsl:choose> <xsl:when test=
          "string-length($text) = '2'"> <xsl:value-of select=
          "$text"/> </xsl:when> <xsl:otherwise> <xsl:variable name=
          "currentHex" select=
          "substring($text,$index,2)"/> <xsl:value-of select=
          "document('char-map.xml')/characterMap/character/hex[.=$currentHex]/../alt/text()"/> <xsl:text>, </xsl:text> <xsl:call-template name=
          "csvGen"> <xsl:with-param name=
          "index" select=
          "1"/> <xsl:with-param name=
          "text" select=
          "substring($text,$index+2)"/> <xsl:with-param name=
          "strLength" select=
          "string-length(substring($text,$index+2))"/> </xsl:call-template> </xsl:otherwise> </xsl:choose> </xsl:template> </xsl:stylesheet>
          


          mapping document from hex to ascii can be like this :

          
          <characterMap> <character> <ascii>A</ascii> <alt>65</alt> <hex>41</hex> </character> ... </characterMap>
          


          which produces the same output ... I liked your way though.

          Thanks.
          • Liv2luv
            Liv2luv
            573 Posts
            ACCEPTED ANSWER

            Re: Conversion of XML file to a comma separated ASCII file

            ‏2011-08-02T14:31:43Z  in response to Liv2luv
            BTW, a node-set with hex stream will be produced by using hexBinary.ffd in the dp:input-mapping.

            Thanks.
        • Liv2luv
          Liv2luv
          573 Posts
          ACCEPTED ANSWER

          Re: Conversion of XML file to a comma separated ASCII file

          ‏2011-08-03T11:02:11Z  in response to HermannSW
          Hi Hermann,

          Further, based on your approach, I've run some tests. At the content length of 15026 bytes and above, my datapower device was reporting a stack overflow error. Decoding base-64 for each character recursively could be the cause.

          Thanks.
          • HermannSW
            HermannSW
            4488 Posts
            ACCEPTED ANSWER

            Re: Conversion of XML file to a comma separated ASCII file

            ‏2011-08-03T15:40:23Z  in response to Liv2luv
            >
            > Further, based on your approach, I've run some tests. At the content length of 15026 bytes and above,
            > my datapower device was reporting a stack overflow error. Decoding base-64 for each character recursively could be the cause.
            >
            Sorry, the solution I posted was the simple one.
            Yes, the recursion is the cause.

            See slide "Doing recursion right" on how the posted stylesheet could be modified:
            http://www-01.ibm.com/support/docview.wss?uid=swg27019119&aid=1#page=5

            Also see slides on "Stylesheet Profiling":
            http://www-01.ibm.com/support/docview.wss?uid=swg27019118&aid=1#page=4

            I do not have the time but would bet that your lookup solution, while working fine, may have some performance penalties.

            On one of the slides I stated
            "Prefer use of extension functions over pure XSLT solutions because of performance"
            and this is still true.
            Extension function dp:radix-convert(_,64,10) should be more efficient than looking up a charmap in XSLT.
            Perhaps you may want to do some profiling?

             
            Hermann<myXsltBlog/>
        • charanchowdary
          charanchowdary
          40 Posts
          ACCEPTED ANSWER

          Re: Conversion of XML file to a comma separated ASCII file

          ‏2011-08-10T19:15:13Z  in response to HermannSW
          Thank you Hermann. It works. However my DP configuration is not able to produce a valid message for the back end AS/400 application over TCP/IP. Here is what I have.

          1. Attached word document <<AS400IntegrationSpec.doc>> has back end AS/400 integration specifications.
          2. I have configured MPG object with

          a) xform action in request rule with below xslt, which is a modified version of above posted asc.xsl to include metadata per above point 1.
          b) static back end as tcp://as400hostname:as400port
          c) Request type as xml and
          d) Response type as non-xml

          <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dp="http://www.datapower.com/extensions" xmlns:func="http://exslt.org/functions" extension-element-prefixes="dp">
          <xsl:output omit-xml-declaration="yes"/>
          <func:function name="func:b64-to-dec">
          <xsl:param name="str"/>
          <xsl:choose>
          <xsl:when test="not($str)"/>
          <xsl:otherwise>
          <func:result select="
          concat(
          dp:radix-convert(dp:substring-base64($str, 1, 1), 64, 10),
          func:b64-to-dec(dp:substring-base64($str, 2))
          )"/>
          </xsl:otherwise>
          </xsl:choose>
          </func:function>
          <xsl:template match="/">
          <xsl:variable name="str">
          <dp:serialize omit-xml-decl="yes" select="."/>
          </xsl:variable>
          <xsl:variable name="XMLByteArray">
          <xsl:value-of select="func:b64-to-dec(dp:encode($str,'base-64'))"/>
          </xsl:variable>
          <xsl:message dp:priority="debug">XMLByteArray: <xsl:value-of select="$XMLByteArray"/>
          </xsl:message>
          <xsl:variable name="XMLByteArrayLength">
          <xsl:value-of select="string-length($XMLByteArray)"/>
          <!-- <xsl:value-of select="dp:variable('var://service/mpgw/request-size')"/> -->
          </xsl:variable>
          <xsl:message dp:priority="debug">XMLByteArrayLength: <xsl:value-of select="$XMLByteArrayLength"/>
          </xsl:message>
          <xsl:value-of select="1"/>
          <xsl:value-of select="$XMLByteArrayLength"/>
          <xsl:value-of select="0"/>
          <xsl:value-of select="$XMLByteArray"/>
          </xsl:template>
          </xsl:stylesheet>

          with above config AS400 application is throwing some internal error as it is unable to process the message sent from DataPower.
          Wondering about what went wrong with above xslt. Any help is welcome.
          • HermannSW
            HermannSW
            4488 Posts
            ACCEPTED ANSWER

            Re: Conversion of XML file to a comma separated ASCII file

            ‏2011-08-11T21:06:05Z  in response to charanchowdary
            > Thank you Hermann. It works. However my DP configuration is not able to produce a valid message
            > for the back end AS/400 application over TCP/IP. Here is what I have.
            >
            > 1. Attached word document <<AS400IntegrationSpec.doc>> has back end AS/400 integration specifications.
            > 2. I have configured MPG object with
            >
            > a) xform action in request rule with below xslt, which is a modified version of above posted asc.xsl to include metadata per above point 1.
            > b) static back end as tcp://as400hostname:as400port
            correct.

            > c) Request type as xml and
            > d) Response type as non-xml
            correct.

            >
            > <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dp="http://www.datapower.com/extensions" xmlns:func="http://exslt.org/functions" extension-element-prefixes="dp">
            > <xsl:output omit-xml-declaration="yes"/>
            ...
            ...
            > <xsl:value-of select="1"/>
            > <xsl:value-of select="$XMLByteArrayLength"/>
            > <xsl:value-of select="0"/>
            > <xsl:value-of select="$XMLByteArray"/>
            > </xsl:template>
            > </xsl:stylesheet>
            >
            > with above config AS400 application is throwing some internal error as it is unable to process the message sent from DataPower.
            > Wondering about what went wrong with above xslt. Any help is welcome.
            >
            Either I or you does not understand your specification document correctly.

            The 4th part of a request by your spec is a "XML Byte Array", the length of which has been specified in 3rd entry "Document Byte Array Length".

            You have to send "bytes" (octets), and not a sequence of decimal numbers like you did if I am right.
            Then you do not need the conversion you asked for at all.
            You should just do <xsl:copy-of select="/" /> and be fine ...

             
            Hermann<myXsltBlog/>
            • charanchowdary
              charanchowdary
              40 Posts
              ACCEPTED ANSWER

              Re: Conversion of XML file to a comma separated ASCII file

              ‏2011-08-11T22:12:41Z  in response to HermannSW
              Thanks Hermann for the response. I certainly didn't understand the AS/400 requirements.

              Below is the updated xsl updated per your comments and the AS/400 app still not able to process the message from DP. Is there anything wrong with the xsl code in bold per the above posted AS/400 Integration spec. Attached java client <<TestIP.java>> is working fine with no issues.

              <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:dp="http://www.datapower.com/extensions" extension-element-prefixes="dp" exclude-result-prefixes="dp">
              <xsl:output omit-xml-declaration="yes"/>
              <xsl:template match="/">
              <xsl:variable name="XMLByteArrayLength">
              <xsl:value-of select="dp:variable('var://service/mpgw/request-size')"/>
              </xsl:variable>
              <xsl:message dp:priority="debug">XMLByteArrayLength: <xsl:value-of select="$XMLByteArrayLength"/>
              </xsl:message>
              <xsl:value-of select="1"/>
              <xsl:value-of select="$XMLByteArrayLength"/>
              <xsl:value-of select="0"/>
              <xsl:copy-of select="/"/>
              </xsl:template>
              </xsl:stylesheet>

              Thanks in advance again.
              • HermannSW
                HermannSW
                4488 Posts
                ACCEPTED ANSWER

                Re: Conversion of XML file to a comma separated ASCII file

                ‏2011-08-14T01:29:16Z  in response to charanchowdary
                Hi,

                > ...
                > Below is the updated xsl updated per your comments and the AS/400 app
                > still not able to process the message from DP. Is there anything wrong
                > with the xsl code in bold per the above posted AS/400 Integration spec.
                > Attached java client <<TestIP.java>> is working fine with no issues.
                > ...
                your Java client is not matching your spec!

                This little stylesheet does what your Java client does (just copy the read XML file):
                <xsl:stylesheet version="1.0"
                  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                >
                  <xsl:template match="/">
                    <xsl:copy-of select="."/>
                  </xsl:template>
                  
                </xsl:stylesheet>
                


                Please give it a try and let us know.

                 
                Hermann <myXsltBlog/>
                Updated on 2014-03-25T03:13:15Z at 2014-03-25T03:13:15Z by iron-man
                • charanchowdary
                  charanchowdary
                  40 Posts
                  ACCEPTED ANSWER

                  Re: Conversion of XML file to a comma separated ASCII file

                  ‏2011-08-14T04:35:40Z  in response to HermannSW
                  Thank you for the response.

                  Tried the xsl (xml copy) posted above and AS/400 didn't like it and threw the same error. Please find the attached document which has TCPMon screenshots of both Java and DataPower (with above posted xsl to simply copy input xml file) transactions with same AS/400 app as target.

                  Java Clinet --> AS/400 transaction:

                  Java client is sending request xml prefixing 12 Bytes of meta data per above posted AS/400 Integration spec( you can see 12 Junk characters in TCPMon) and AS/400 is returning the desired response xml prefixing 12 Bytes of meta data per above posted AS/400 Integration spec( you can see 12 Junk characters in TCPMon).

                  DataPower Clinet --> AS/400 transaction:

                  DataPower is sending request xml and AS/400 is returning the error response xml prefixing 12 Bytes of meta data per above posted AS/400 Integration spec( you can see 12 Junk characters in TCPMon).

                  Per my understanding, the above posted xsl(copy xml) should be updated to output metadata (per AS/400 Integration spec
                  ) which should be equivalent of below 3 lines of java client code.

                  out.writeInt( 1 );
                  out.writeInt( xml.length ) ;
                  out.writeInt( 0 ) ;

                  Thanks in advance again...
                  • HermannSW
                    HermannSW
                    4488 Posts
                    ACCEPTED ANSWER

                    Re: Conversion of XML file to a comma separated ASCII file

                    ‏2011-08-14T14:42:29Z  in response to charanchowdary
                    HI,

                    > ...
                    > Java client is sending request xml prefixing 12 Bytes of meta data per above posted AS/400 Integration spec( you can see 12 Junk characters in TCPMon) and AS/400 is returning the desired response xml prefixing 12 Bytes of meta data per above posted AS/400 Integration spec( you can see 12 Junk characters in TCPMon).
                    >
                    > DataPower Clinet --> AS/400 transaction:
                    >
                    > DataPower is sending request xml and AS/400 is returning the error response xml prefixing 12 Bytes of meta data per above posted AS/400 Integration spec( you can see 12 Junk characters in TCPMon).
                    >
                    > Per my understanding, the above posted xsl(copy xml) should be updated to output metadata (per AS/400 Integration spec
                    > ) which should be equivalent of below 3 lines of java client code.
                    >
                    > out.writeInt( 1 );
                    > out.writeInt( xml.length ) ;
                    > out.writeInt( 0 ) ;
                    >
                    tcpmon showed that your spec was completely correct (I misread what your Java code does).

                    You cannot send values like 0x00 bytes from within a stylesheet.
                    You do need binary transformations to do so.

                    You may either use WTX.
                    Or below solution based on the FFD shipped with the DataPower box (store:///pkcs7-convert-input.ffd).
                    See posting DataPower "binary" node type (binaryNode) for details on below solution:
                    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14572518#14572518

                    Here you can see stylesheet "AS400.xsl" in action:
                    
                    $ echo 
                    "<a>258</a>" | coproc2 AS400.xsl - http:
                    //dp3-l3:2223 | od -Ax -tcx1 000000  \0  \0  \0 001  \0  \0  \0  \n  \0  \0  \0  \0   <a>   2 00 00 00 01 00 00 00 0a 00 00 00 00 3c 61 3e 32 000010   5   8   <   / a   > 35 38 3c 2f 61 3e 000016 $
                    


                    And this is your stylesheet (also attached for download), thanks for this nice task:
                    
                    $ cat AS400.xsl <!-- Convert input XML to AS/400 data stream: 00 00 00 01  32bitlen  00 00 00 00  XMLdata --> <xsl:stylesheet version=
                    "1.0" xmlns:xsl=
                    "http://www.w3.org/1999/XSL/Transform" xmlns:func=
                    "http://exslt.org/functions" xmlns:dp=
                    "http://www.datapower.com/extensions" extension-element-prefixes=
                    "dp" > <dp:output-mapping href=
                    "store:///pkcs7-convert-input.ffd" type=
                    "ffd"/> <!-- This FFD converts the XML tree like 
                    
                    this to binary output:   <object> <message>***binary data***</message> </object> -->   <xsl:output omit-xml-declaration=
                    "yes" />   <!-- convert unsigned number to base64 encoded 32bit integer.   258 --> AAABAg==  (= 00 00 01 02) --> <func:function name=
                    "func:uint2b64"> <xsl:param name=
                    "u"/> <func:result select=
                    "dp:substring-base64( dp:radix-convert($u+4294967296,10,64), 2 )
                    "/> </func:function>     <xsl:template match=
                    "/"> <!-- Serialize XML input --> <xsl:variable name=
                    "serialized"> <dp:serialize select=
                    "." omit-xml-decl=
                    "yes"/> </xsl:variable>   <!-- determine AS/400 encoding --> <xsl:variable name=
                    "b64" select=
                    "dp:concat-base64( dp:concat-base64( func:uint2b64(1), func:uint2b64(string-length($serialized)) ), dp:concat-base64( func:uint2b64(0), dp:encode($serialized,
                    'base-64') ) )
                    " />   <!-- prepare binaryNode 
                    
                    return 
                    
                    for dp:output-mapping --> <object> <message> <xsl:copy-of select=
                    "dp:binary-decode($b64)" /> </message> </object> </xsl:template> </xsl:stylesheet> $
                    


                     
                    Hermann<myXsltBlog/>
                    • charanchowdary
                      charanchowdary
                      40 Posts
                      ACCEPTED ANSWER

                      Re: Conversion of XML file to a comma separated ASCII file

                      ‏2011-08-14T22:29:45Z  in response to HermannSW
                      Thank you Hermann. The above posted AS400.xsl worked like a charm and the AS400 app is now returning the desired response with a prefix of 12 Bytes of metadata(per integration spec) which now I need to convert it to pure xml part of response rule processing. Could you share the steps to do the same. I think Response type of MPG object should be Non-XML.

                      Thanks in advance again!!
                      • HermannSW
                        HermannSW
                        4488 Posts
                        ACCEPTED ANSWER

                        Re: Conversion of XML file to a comma separated ASCII file

                        ‏2011-08-15T18:07:10Z  in response to charanchowdary
                        Hi,

                        > Thank you Hermann. The above posted AS400.xsl worked like a charm
                        cool.

                        > and the AS400 app is now returning the desired response with a prefix of 12 Bytes of metadata(per integration spec)
                        > which now I need to convert it to pure xml part of response rule processing.
                        > Could you share the steps to do the same.
                        See below.

                        > I think Response type of MPG object should be Non-XML.
                        >
                        Correct.

                        I captured the output of AS400.xsl for the sample input from previous posting ("<a>258</a>") in AS400.dat.
                        Then I created 4 testcases failing the implemented validation in new stylesheet 004SA.xsl.
                        
                        $ od -Ax -tx1 AS400.dat 000000 00 00 00 01 00 00 00 0a 00 00 00 00 3c 61 3e 32 000010 35 38 3c 2f 61 3e 000016 $ od -Ax -tx1 AS400.1.dat 000000 00 00 00 02 00 00 00 0a 00 00 00 00 3c 61 3e 32 000010 35 38 3c 2f 61 3e 000016 $ od -Ax -tx1 AS400.2.dat 000000 00 00 00 01 00 00 00 0a 00 00 00 00 3c 61 3e 32 000010 35 fc 3c 2f 61 3e 000016 $ od -Ax -tx1 AS400.3.dat 000000 00 00 00 01 00 00 00 09 00 00 00 00 3c 61 3e 32 000010 35 38 3c 2f 61 3e 000016 $ od -Ax -tx1 AS400.4.dat 000000 00 00 00 01 00 00 00 0a 00 00 00 01 3c 61 3e 32 000010 35 38 3c 2f 61 3e 000016 $ od -Ax -tx1 AS400.5.dat 000000 00 00 00 01 00 00 00 0a 00 00 00 00 3c 61 3e 32 000010 35 38 3c 20 61 3e 000016 $
                        


                        These are the error messages logged for input files
                        • AS400.1.dat: xsl:message terminate=yes value='1st integer not 1'
                        • AS400.2.dat: Valid base64 passed to dp:decode resulted in non UTF8: 'PGE+MjX8PC9hPg=='
                        • AS400.3.dat: xsl:message terminate=yes value='XML length does not match'
                        • AS400.4.dat: xsl:message terminate=yes value='3rd integer not 0'
                        • AS400.5.dat: xsl:message terminate=yes value='Non-XML payload'

                        Of course you can skip the validation steps 1, 3, 4 and 5 in stylesheet 004SA.xsl (also attached) if you trust getting correct XML messages from backend:
                        
                        <!-- Extract XMLdata from AS/400 data stream: 00 00 00 01  32bitlen  00 00 00 00  XMLdata --> <xsl:stylesheet version=
                        "1.0" xmlns:xsl=
                        "http://www.w3.org/1999/XSL/Transform" xmlns:func=
                        "http://exslt.org/functions" xmlns:dp=
                        "http://www.datapower.com/extensions" extension-element-prefixes=
                        "dp" > <dp:input-mapping href=
                        "store:///pkcs7-convert-input.ffd" type=
                        "ffd"/> <!-- This FFD converts the input into an XML tree like 
                        
                        this:   <object> <message>***binary data***</message> </object> -->   <xsl:output omit-xml-declaration=
                        "yes" />   <xsl:template match=
                        "/"> <!-- complete input as base64 encoded string --> <xsl:variable name=
                        "all" select=
                        "dp:binary-encode(/object/message)" />   <!-- 1 --> <xsl:
                        
                        if test=
                        "dp:radix-convert(dp:substring-base64($all,1,4),64,10)!=1"> <xsl:message terminate=
                        "yes">1st integer not 1</xsl:message> </xsl:if>   <!-- 2 --> <!-- serialized XML input, inclusive UTF-8 validation --> <xsl:variable name=
                        "serialized" select=
                        "dp:decode(dp:substring-base64($all,13),'base-64')" />   <!-- 3 --> <xsl:
                        
                        if test=
                        "dp:radix-convert(dp:substring-base64($all,5,4),64,10) != string-length($serialized)
                        "> <xsl:message terminate=
                        "yes">XML length does not match</xsl:message> </xsl:if>   <!-- 4 --> <xsl:
                        
                        if test=
                        "dp:radix-convert(dp:substring-base64($all,9,4),64,10)!=0"> <xsl:message terminate=
                        "yes">3rd integer not 0</xsl:message> </xsl:if>   <xsl:variable name=
                        "xml"> <xsl:copy-of select=
                        "dp:parse($serialized)" /> </xsl:variable>   <!-- 5 --> <xsl:
                        
                        if test=
                        "not($xml/*)"> <xsl:message terminate=
                        "yes">Non-XML payload</xsl:message> </xsl:if>   <xsl:copy-of select=
                        "$xml"/> </xsl:template> </xsl:stylesheet>
                        

                        coproc2 output for the five testcases:
                        
                        $ coproc2 004SA.xsl AS400.dat http:
                        //dp3-l3:2223/nonxml -s; echo <a>258</a> $ coproc2 004SA.xsl AS400.1.dat http:
                        //dp3-l3:2223/nonxml -s; echo <?xml version=
                        "1.0" encoding=
                        "UTF-8"?> <env:Envelope xmlns:env=
                        "http://schemas.xmlsoap.org/soap/envelope/"><env:Body><env:Fault><faultcode>env:Client</faultcode><faultstring>Internal Error (from client)</faultstring></env:Fault></env:Body></env:Envelope> $ coproc2 004SA.xsl AS400.2.dat http:
                        //dp3-l3:2223/nonxml -s; echo <?xml version=
                        "1.0" encoding=
                        "UTF-8"?> <env:Envelope xmlns:env=
                        "http://schemas.xmlsoap.org/soap/envelope/"><env:Body><env:Fault><faultcode>env:Client</faultcode><faultstring>Internal Error (from client)</faultstring></env:Fault></env:Body></env:Envelope> $ coproc2 004SA.xsl AS400.3.dat http:
                        //dp3-l3:2223/nonxml -s; echo <?xml version=
                        "1.0" encoding=
                        "UTF-8"?> <env:Envelope xmlns:env=
                        "http://schemas.xmlsoap.org/soap/envelope/"><env:Body><env:Fault><faultcode>env:Client</faultcode><faultstring>Internal Error (from client)</faultstring></env:Fault></env:Body></env:Envelope> $ coproc2 004SA.xsl AS400.4.dat http:
                        //dp3-l3:2223/nonxml -s; echo <?xml version=
                        "1.0" encoding=
                        "UTF-8"?> <env:Envelope xmlns:env=
                        "http://schemas.xmlsoap.org/soap/envelope/"><env:Body><env:Fault><faultcode>env:Client</faultcode><faultstring>Internal Error (from client)</faultstring></env:Fault></env:Body></env:Envelope> $ coproc2 004SA.xsl AS400.5.dat http:
                        //dp3-l3:2223/nonxml -s; echo <?xml version=
                        "1.0" encoding=
                        "UTF-8"?> <env:Envelope xmlns:env=
                        "http://schemas.xmlsoap.org/soap/envelope/"><env:Body><env:Fault><faultcode>env:Client</faultcode><faultstring>Internal Error (from client)</faultstring></env:Fault></env:Body></env:Envelope> $
                        


                         
                        Hermann<myXsltBlog/>
                        • charanchowdary
                          charanchowdary
                          40 Posts
                          ACCEPTED ANSWER

                          Re: Conversion of XML file to a comma separated ASCII file

                          ‏2011-08-15T21:08:12Z  in response to HermannSW
                          Thank you so much for the quick response. Above style sheet looks good to me though we ran in to below Network error during binary response processing for this I have opened a PMR as even the simple text to XML wrapping scenario per the technote @ http://www-01.ibm.com/support/docview.wss?uid=swg21321379 is also resulting in below network error.

                          10.93.19.81 0x80c00008 mpgw (test): rule (test-sp-response-rule): implied action Read input as binary data failed: Network error

                          Attached <<debug-syslog-10455584.doc>> word document has the debug level sys log entries for the transaction id # 10455584 in question along with the probe screenshot which shows the received binary data from the back end AS/400 application.

                          Thank you again for your assistance!!!!
                          • HermannSW
                            HermannSW
                            4488 Posts
                            ACCEPTED ANSWER

                            Re: Conversion of XML file to a comma separated ASCII file

                            ‏2011-08-15T21:24:53Z  in response to charanchowdary
                            > Thank you so much for the quick response. Above style sheet looks good to me though we ran in to below
                            > Network error during binary response processing for this I have opened a PMR as even the simple text to
                            > XML wrapping scenario per the technote @ http://www-01.ibm.com/support/docview.wss?uid=swg21321379 is
                            > also resulting in below network error.
                            >
                            > 10.93.19.81 0x80c00008 mpgw (test): rule (test-sp-response-rule): implied action Read input as binary data failed: Network error
                            >
                            > Attached <<debug-syslog-10455584.doc>> word document has the debug level sys log entries for the transaction id # 10455584 in question along with the probe screenshot which shows the received binary data from the back end AS/400 application.
                            >

                            Below two lines from your syslog show that DataPower is trying to parse your response message as XML.
                            But your response is binary.

                            What you need to have in place is:
                            • Response Type "Non-XML"
                            • Have stylesheet "004SA.xsl" in a "Binary Transform Action"

                            The parse error seems to indicate that you have a "normal" transform action in response rule ...
                            
                            ... 14:23:06        xmlparse        debug   10455584        response        10.93.19.81     0x80e003a6      mpgw (test): Parsing document: 
                            'http://161.159.103.117:4201/AWD-V1' 14:23:06  multistep       error   10455584        response        10.93.19.81     0x80c00008      mpgw (test): rule (test-sp-response-rule): implied action Read input as binary data failed: Network error ...
                            


                             
                            Hermann<myXsltBlog/>
                            • charanchowdary
                              charanchowdary
                              40 Posts
                              ACCEPTED ANSWER

                              Re: Conversion of XML file to a comma separated ASCII file

                              ‏2011-08-15T21:36:50Z  in response to HermannSW
                              > What you need to have in place is:
                              > * Response Type "Non-XML"
                              > * Have stylesheet "004SA.xsl" in a "Binary Transform Action"

                              I actually have my Response Type as "Non-XML" and I do have the "004SA.xsl" in a "Binary Transform Action" per the attached probe export <<test-probe-export-capture.zip>> .

                              Thank you again!!!.
                              • HermannSW
                                HermannSW
                                4488 Posts
                                ACCEPTED ANSWER

                                Re: Conversion of XML file to a comma separated ASCII file

                                ‏2011-08-16T09:23:35Z  in response to charanchowdary
                                Hi,
                                >
                                > I actually have my Response Type as "Non-XML" and I do have the "004SA.xsl" in a "Binary Transform Action" per the attached probe export <<test-probe-export-capture.zip>> .
                                >

                                I could not immediately see what is wrong with your configuration.

                                So I just built the attached MPGW from scratch making use of my previously posted stylesheets AS400.xsl and 004SA.xsl.

                                My DataPower box dp3-l3 service listens on port 4445 and goes to backend tcp://dp4-l3:4444.

                                I use netcat tool as TCP server.
                                First I tested the setup.
                                This is TCP server emulation on dp4-l3 returning "test" received from client:
                                
                                [stammw@dp4-l3 ~]$ nc -l 4444 <AS400.dat | od -tcx1 0000000   t   e   s   t  \n 74 65 73 74 0a 0000005 [stammw@dp4-l3 ~]$
                                


                                This is the (test) client (receiving the AS400.dat response):
                                
                                $ echo 
                                "test" | nc dp4-l3 4444 - | od -tcx1 0000000  \0  \0  \0 001  \0  \0  \0  \n  \0  \0  \0  \0   <a>   2 00 00 00 01 00 00 00 0a 00 00 00 00 3c 61 3e 32 0000020   5   8   <   / a   > 35 38 3c 2f 61 3e 0000026 $
                                


                                Now after this works here is TCP server on dp4-l3 again (with request received from DataPower):
                                
                                [stammw@dp4-l3 ~]$ nc -l 4444 <AS400.dat | od -tcx1 0000000  \0  \0  \0 001  \0  \0  \0  \n  \0  \0  \0  \0   <   b   >   1 00 00 00 01 00 00 00 0a 00 00 00 00 3c 62 3e 31 0000020   2   3   <   / b   > 32 33 3c 2f 62 3e 0000026 [stammw@dp4-l3 ~]$
                                


                                And this is the curl client going through DataPower service to TCP server receiving the
                                decoded AS400.dat XML message (different to the message sent!):
                                
                                $ echo 
                                "<b>123</b>" | curl --data-binary @- http:
                                //dp3-l3:4445; echo <a>258</a> $
                                


                                Now you have to look what is wrong in your configuration ...

                                 
                                Hermann<myXsltBlog/>

                                Attachments

                                • charanchowdary
                                  charanchowdary
                                  40 Posts
                                  ACCEPTED ANSWER

                                  Re: Conversion of XML file to a comma separated ASCII file

                                  ‏2011-08-16T15:41:04Z  in response to HermannSW
                                  1. I verified both of our configs are similar and they both are looking good.
                                  2. a) I did import your MPGW(4444)to my XI50-->9235-4BX device running on 3.8.2.3 (I don't think this should have any compatibility issues as source 3.8.2.5 is on bit higher level than target 3.8.2.3)
                                  b)Only changed the back side URL to point it to AS/400 application.

                                  and when tested I see the same network error that I see with my MPGW. Below are the service variable values related to the error on step 0 of the response leg. Attaching the probe Export Capture <<4444-probe-ExportCapture.zip>>.

                                  var://service/error-code string 0x00530001
                                  var://service/error-message string 'Network error'
                                  var://service/error-subcode string 0x00530001

                                  3. PMR support team is analyzing the pcap files currently.

                                  Thank you again!!!
                                  • HermannSW
                                    HermannSW
                                    4488 Posts
                                    ACCEPTED ANSWER

                                    Re: Conversion of XML file to a comma separated ASCII file

                                    ‏2011-08-16T17:50:29Z  in response to charanchowdary
                                    Hi,

                                    the Probe isn't helpful here.

                                    If you want to do next step of investigation you should
                                    • setup the config against a AS/400 emulating netcat server as I did (that will work as I showed)
                                    • setup as you already have against real AS/400 server.

                                    For both scenarios you should take a DataPower packet capture on the interface going to AS/400 (emulation or real server).

                                    Then you will be able to identify the difference by (Wireshark) "Follow TCP stream" in both pcaps.
                                    Since AS/400 emulation works (as shown above) we already know what data comes back to DataPower.
                                    Your real AS/400 system seems to do something different (not neccesarily data, but FINs or ACKs).

                                     
                                    Hermann<myXsltBlog/>
                                    • charanchowdary
                                      charanchowdary
                                      40 Posts
                                      ACCEPTED ANSWER

                                      Re: Conversion of XML file to a comma separated ASCII file

                                      ‏2011-08-25T21:10:44Z  in response to HermannSW
                                      Here is the shocking nice note which we just received from IBM Data Power support team for the PMR we raised 2 weeks ago.

                                      As stated during the call, after further discussion with our development team, for TCP, the current message type request/response supported in DataPower is XML.
                                      Please feel free to contact me for any follow-up questions.
                                      DataPower is throwing Network error (with error code 0x00530001) only once Back Side Timeout is reached and DataPower can't close the TCP connection as it receives the TCP response from the backend AS/400 application.

                                      Below is the tcp stream for the communication between DataPower and AS/400 application over TCP where you can see a error response with java.io.EOFException (underlined) too from the as/400 application due to non closure of the tcp connection along with the desired response (bold). request is in italics.

                                      ............<DST xml:lang="en-US">
                                      <jobName version="1.0">SRVInit</jobName>
                                      <trace>0</trace>
                                      <readable>N</readable>
                                      <AWD>
                                      <userID>PLATFRM1</userID>
                                      <password>12345678</password>
                                      <config dictionary="N" userPrivileges="N" options="Y"/>
                                      </AWD>
                                      </DST>............<?xml version="1.0" ?><DST xml:lang="en-US"><jobName version="1.0">SRVInit</jobName><trace>0</trace><readable>N</readable><AWD><userID>PLATFRM1</userID><config dictionary="N" userPrivileges="N" options="Y"/><options><customScreenPath>\\HOUPFNAS01\TSTSHARE\LAPPS\CSDPTEST</customScreenPath><faxServer>Y</faxServer><printServer>N</printServer></options></AWD><jobReturn><process>com.dstsystems.awd.ns2.jobManager.ke.tasks.AWDPost</process>0<description></description></jobReturn></DST>............<?xml version="1.0" ?><DST><error>RCVIPREQ<text>An IO Error occured receiving request java.io.EOFException.</text><task>AWD 3.2 XML Services</task></error></DST>

                                      As a work around I have configured an error rule with below actions and able to extract and send the xml response with http status code as 200 though it's not ideal solution until IBM comes up with a solution in their future releases to support Non-xml over TCP.

                                      1. Match rule as Error Code 0x00530001
                                      2. Transform Binary Action with xsl same as one that i have in response rule with INPUT as INPUT and OUTPUT as var1
                                      3. Transform action with Identity xsl with INPUT as var1 and OUTPUT as var2
                                      4. Set Variable action to set var://service/error-protocol-response to 200
                                      5. Set Variable action to set var://service/error-protocol-reason-phrase to OK

                                      Pls let me know if you guys have any other thoughts/recommandations.

                                      Thank you Hermann for all your great responses!!!!
                                      • HermannSW
                                        HermannSW
                                        4488 Posts
                                        ACCEPTED ANSWER

                                        Re: Conversion of XML file to a comma separated ASCII file

                                        ‏2011-08-28T12:32:54Z  in response to charanchowdary
                                        Hi,

                                        I have not had the time to read this before.
                                        And now I am on the train for a 1 week vacation with my KS360 cell phone but without access to the IBM network and my Datapower boxes in Böblingen lab.
                                        So you found a way to deal with processing the error response.
                                        I am not sure whether this is all you need.

                                        If the Java IO error occurs due to the fact that the connection to the AS/400 does not get closed I probably have a solution for you ...
                                        • HermannSW
                                          HermannSW
                                          4488 Posts
                                          ACCEPTED ANSWER

                                          Re: Conversion of XML file to a comma separated ASCII file

                                          ‏2011-08-28T12:53:11Z  in response to HermannSW
                                          Terminating the connection from DataPower side (sending FIN ACK) is not directly possible.
                                          But you can set frontside timeout of DataPower service to eg. 1second and use this as a workaround.
                                          1 second after the complete message has been sent and nothing new has been received from the frontside DataPower will timeout and send FIN ACK to the backside, exactly what is needed.
                                          (sorry for splitting messages, Opera Mini 6 browser is really cool, but text input is bound to SMS size by my cell phone)
                                          Hermann.
                                          • charanchowdary
                                            charanchowdary
                                            40 Posts
                                            ACCEPTED ANSWER

                                            Re: Conversion of XML file to a comma separated ASCII file

                                            ‏2011-08-29T06:20:56Z  in response to HermannSW
                                            Thank you Hermann for the response. I am fine with the Workaround with appropriate front end and back end timeout settings. I Just wanted to update you on the status and check for a better workaround/solution than what I have currently.

                                            Thank you again for all the support.