Topic
  • 11 replies
  • Latest Post - ‏2011-08-24T22:26:45Z by dpdeveloper
bionicl
bionicl
13 Posts

Pinned topic validate multiple xml in a zip

‏2010-11-15T07:55:35Z |
Hi all,

im seeking advice with regards to the following scenario.

1)client application will do a batch processing and zip up multiple xml into 1 zip file (call abc.zip) and place it into a specific directory
2)dp will do a ftp poll on that directory and retrieve the abc.zip file
3)the use of dp here will be to validate each individual xml in the zip file for xml threats (no schema validation is required)
4)upon success/completion, dp will rename the zip file as abc_processed.zip and place into another directory.

i'm wondering how to go about implementing this? as in does dp has such capability to do so?

thanks and sorry for the trouble!

:)
Updated on 2011-08-24T22:26:45Z at 2011-08-24T22:26:45Z by dpdeveloper
  • mmatamoros
    mmatamoros
    60 Posts

    Re: validate multiple xml in a zip

    ‏2010-11-17T17:45:06Z  
    what you can do is the following:

    • After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    • The you do a dp:parse to get the XML files from the base64 string
    • Do the validations
    • Then process back the zip file with deflate and send it to the backend
  • HermannSW
    HermannSW
    4723 Posts

    Re: validate multiple xml in a zip

    ‏2010-11-18T17:39:20Z  
    what you can do is the following:

    • After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    • The you do a dp:parse to get the XML files from the base64 string
    • Do the validations
    • Then process back the zip file with deflate and send it to the backend
    > what you can do is the following:
    >
    > - After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    > - The you do a dp:parse to get the XML files from the base64 string
    > - Do the validations
    > - Then process back the zip file with deflate and send it to the backend

    .zip is a compressed archive format, not sure how you can inflate that ...
    What I would propose instead is the following two step approach:
    1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701

    2) extract the files from that attachment (the .zip file) using "var://local/attachment-manifest",
    see chapter "Attachment protocol" from XSL Developers Accelerator guide:
    http://www-01.ibm.com/software/integration/datapower/library/documentation/v3.8.0/XS40.html

    Hermann.
  • mmatamoros
    mmatamoros
    60 Posts

    Re: validate multiple xml in a zip

    ‏2010-11-18T17:44:56Z  
    • HermannSW
    • ‏2010-11-18T17:39:20Z
    > what you can do is the following:
    >
    > - After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    > - The you do a dp:parse to get the XML files from the base64 string
    > - Do the validations
    > - Then process back the zip file with deflate and send it to the backend

    .zip is a compressed archive format, not sure how you can inflate that ...
    What I would propose instead is the following two step approach:
    1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701

    2) extract the files from that attachment (the .zip file) using "var://local/attachment-manifest",
    see chapter "Attachment protocol" from XSL Developers Accelerator guide:
    http://www-01.ibm.com/software/integration/datapower/library/documentation/v3.8.0/XS40.html

    Hermann.
    As per the Extension Functions Catalog the extension inflate() says:

    Decompresses a Base64-encoded string and returns a decoded version of the string.

    dp:inflate(text, algorithm)

    Algorithm

    inflate
    (Default) Decompresses the string with the algorithm that is
    described in RFC 1951. The format is LZ77 with Huffman encoding
    (Deutsch with an Adler checksum).

    gzip Decompresses the string with the algorithm that is described in
    RFC 1952. The format is LZ77 with a 32-bit checksum. (Deutsch
    without an Adler checksum).
    compress

    Decompresses the string with the algorithm that is described in
    RFC 1950. The format is adaptive LZW.
  • HermannSW
    HermannSW
    4723 Posts

    Re: validate multiple xml in a zip

    ‏2010-11-18T19:31:48Z  
    As per the Extension Functions Catalog the extension inflate() says:

    Decompresses a Base64-encoded string and returns a decoded version of the string.

    dp:inflate(text, algorithm)

    Algorithm

    inflate
    (Default) Decompresses the string with the algorithm that is
    described in RFC 1951. The format is LZ77 with Huffman encoding
    (Deutsch with an Adler checksum).

    gzip Decompresses the string with the algorithm that is described in
    RFC 1952. The format is LZ77 with a 32-bit checksum. (Deutsch
    without an Adler checksum).
    compress

    Decompresses the string with the algorithm that is described in
    RFC 1950. The format is adaptive LZW.
    > As per the Extension Functions Catalog the extension inflate() says:
    >
    > Decompresses a Base64-encoded string and returns a decoded version of the string.
    >
    > dp:inflate(text, algorithm)
    >

    Try it and you will see the problem.
    dp:inflate() returns the decoded "string" which has been compressed and then base64 encoded before.

    But a .zip file is an "archive" format.
    A .zip file typically contains more than one file.
    Not sure what dp:inflate() would give you, definitely only one "string", but how do you want to extract the contained files from that?
    From DataPower XSL Accelerator Developers Guide:

    Attachment protocol
    The fetch, results, and results-async actions support the attachment protocol.
    This protocol identifies a SOAP attachment and has the following format:
    attachment://context/cid:content_id[?query_param_1&query_param_2]

    The fetch action supports the following query parameters:
    • Encode=base64
    • Compress=gzip
    • Archive=tar or Archive=zip
    • Filename=file_name
    The results and results-async actions support the following query parameters:
    • Decode=base64 or Decode=hexbin
    • Compress=gzip
    • Archive=tar or Archive=zip
    • Filename=file_name
    • Parsable=true
    • Manifest=true

    If you want to get an idea what you can do with the attachment protocol you might want see this slide:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119&aid=1#page=8

    Hermann.
  • HermannSW
    HermannSW
    4723 Posts

    Re: validate multiple xml in a zip

    ‏2010-11-18T20:16:56Z  
    • HermannSW
    • ‏2010-11-18T17:39:20Z
    > what you can do is the following:
    >
    > - After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    > - The you do a dp:parse to get the XML files from the base64 string
    > - Do the validations
    > - Then process back the zip file with deflate and send it to the backend

    .zip is a compressed archive format, not sure how you can inflate that ...
    What I would propose instead is the following two step approach:
    1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701

    2) extract the files from that attachment (the .zip file) using "var://local/attachment-manifest",
    see chapter "Attachment protocol" from XSL Developers Accelerator guide:
    http://www-01.ibm.com/software/integration/datapower/library/documentation/v3.8.0/XS40.html

    Hermann.
    > ...
    > What I would propose instead is the following two step approach:
    > 1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
    > https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701
    > ...

    I found an even better posting of Steven Loscalpio back from 2007:
    "Re: Creating a SOAP attachment from binary file polled from FTP"
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14140244&#14140244

    The posting subject matches really good what bionici asked for above:
    > 2) dp will do a ftp poll on that directory and retrieve the abc.zip file
    Find attached a 3.7.3 export of XML FW makeSwa listening on port 9999.

    Here you can see it in action:
    
    $ cat testdata abtdaa $ od -Ax -tx1 testdata 000000 61 62 03 74 64 61 01 61 0a 000009 $ curl -d @testdata http:
    //dp1-l3.boeblingen.de.ibm.com:9999     --1d72d876-84e0-4877-a946-7a76566d083e Content-Type: text/xml; charset=UTF-8 Content-Transfer-Encoding: binary   <?xml version=
    "1.0" encoding=
    "UTF-8"?> <env:Envelope xmlns:dp=
    "http://www.datapower.com/extensions" xmlns:env=
    "http://schemas.xmlsoap.org/soap/envelope/"><env:Body> WhateverYouWant </env:Body></env:Envelope> --1d72d876-84e0-4877-a946-7a76566d083e Content-ID: <backup.zip> Content-Transfer-Encoding: binary Content-Type: application/x-www-form-urlencoded   abtdaa --1d72d876-84e0-4877-a946-7a76566d083e-- $ $ cat bin.xsl <?xml version=
    "1.0" encoding=
    "utf-8"?> <xsl:stylesheet xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" version=
    "1.0">   <dp:input-mapping href=
    "bin.ffd" type=
    "ffd"/>   <xsl:output method=
    "xml"/>   <xsl:template match=
    "/"> <env:Envelope xmlns:env=
    "http://schemas.xmlsoap.org/soap/envelope/"> <env:Body> WhateverYouWant </env:Body> </env:Envelope> </xsl:template>   </xsl:stylesheet> $ $ cat bin.ffd <?xml version=
    "1.0" encoding=
    "utf-8"?> <File name=
    "file"> <Field name=
    "binaryNode" type=
    "binaryNode"/> </File> $
    


    Hermann.

    Attachments

  • bionicl
    bionicl
    13 Posts

    Re: validate multiple xml in a zip

    ‏2010-11-21T14:52:48Z  
    • HermannSW
    • ‏2010-11-18T20:16:56Z
    > ...
    > What I would propose instead is the following two step approach:
    > 1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
    > https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701
    > ...

    I found an even better posting of Steven Loscalpio back from 2007:
    "Re: Creating a SOAP attachment from binary file polled from FTP"
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14140244&#14140244

    The posting subject matches really good what bionici asked for above:
    > 2) dp will do a ftp poll on that directory and retrieve the abc.zip file
    Find attached a 3.7.3 export of XML FW makeSwa listening on port 9999.

    Here you can see it in action:
    <pre class="jive-pre"> $ cat testdata abtdaa $ od -Ax -tx1 testdata 000000 61 62 03 74 64 61 01 61 0a 000009 $ curl -d @testdata http: //dp1-l3.boeblingen.de.ibm.com:9999 --1d72d876-84e0-4877-a946-7a76566d083e Content-Type: text/xml; charset=UTF-8 Content-Transfer-Encoding: binary <?xml version= "1.0" encoding= "UTF-8"?> <env:Envelope xmlns:dp= "http://www.datapower.com/extensions" xmlns:env= "http://schemas.xmlsoap.org/soap/envelope/"><env:Body> WhateverYouWant </env:Body></env:Envelope> --1d72d876-84e0-4877-a946-7a76566d083e Content-ID: <backup.zip> Content-Transfer-Encoding: binary Content-Type: application/x-www-form-urlencoded abtdaa --1d72d876-84e0-4877-a946-7a76566d083e-- $ $ cat bin.xsl <?xml version= "1.0" encoding= "utf-8"?> <xsl:stylesheet xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" version= "1.0"> <dp:input-mapping href= "bin.ffd" type= "ffd"/> <xsl:output method= "xml"/> <xsl:template match= "/"> <env:Envelope xmlns:env= "http://schemas.xmlsoap.org/soap/envelope/"> <env:Body> WhateverYouWant </env:Body> </env:Envelope> </xsl:template> </xsl:stylesheet> $ $ cat bin.ffd <?xml version= "1.0" encoding= "utf-8"?> <File name= "file"> <Field name= "binaryNode" type= "binaryNode"/> </File> $ </pre>

    Hermann.
    alright guys. thanks alot for all your responses!

    i will give it a try. keep you guys updated ;)
  • dpdeveloper
    dpdeveloper
    111 Posts

    Re: validate multiple xml in a zip

    ‏2011-08-23T21:21:52Z  
    • HermannSW
    • ‏2010-11-18T17:39:20Z
    > what you can do is the following:
    >
    > - After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    > - The you do a dp:parse to get the XML files from the base64 string
    > - Do the validations
    > - Then process back the zip file with deflate and send it to the backend

    .zip is a compressed archive format, not sure how you can inflate that ...
    What I would propose instead is the following two step approach:
    1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701

    2) extract the files from that attachment (the .zip file) using "var://local/attachment-manifest",
    see chapter "Attachment protocol" from XSL Developers Accelerator guide:
    http://www-01.ibm.com/software/integration/datapower/library/documentation/v3.8.0/XS40.html

    Hermann.
    We have a requirement where zip files will be copied into datapower and we have extract the files and copy each file. Is it possible to do this in DP?

    Tried to using the zip file you have provided, but how can we extract the file from attachment?
  • HermannSW
    HermannSW
    4723 Posts

    Re: validate multiple xml in a zip

    ‏2011-08-23T21:35:21Z  
    We have a requirement where zip files will be copied into datapower and we have extract the files and copy each file. Is it possible to do this in DP?

    Tried to using the zip file you have provided, but how can we extract the file from attachment?
    > We have a requirement where zip files will be copied into datapower and we have extract the files and copy each file. Is it possible to do this in DP?
    >
    Yes.

    > Tried to using the zip file you have provided, but how can we extract the file from attachment?
    >
    See below template code snippet.

    It extracts file "filename-you-want-to-extract" from the zip file which is attached under Content-ID "backup.zip" of a SOAP with attachment file as binaryNode.

    The step after that attaches the extracted (Non-XML) file as new attachment (just as a sample).
    You could work with the extrcated binary file in XSLT also by accessing it like
    "dp:binary-encode(tmp/result/binary/child::node())"

    
    ... <xsl:variable name=
    "filename" select=
    "'filename-you-want-to.extract')"/>   <xsl:variable name=
    "tmp"> <dp:url-open target=
    "{concat('cid:backup.zip?Archive=zip&amp;Filename=',$filename)}" response=
    "binaryNode" /> </xsl:variable>   <dp:url-open target=
    "{concat('attachment://out/cid:',$filename)}" response=
    "ignore" resolve-mode=
    "swa"> <xsl:copy-of select=
    "$tmp/result/binary/child::node()"/> </dp:url-open> ...
    

    Hermann<myXsltBlog/>
  • dpdeveloper
    dpdeveloper
    111 Posts

    Re: validate multiple xml in a zip

    ‏2011-08-23T22:27:41Z  
    • HermannSW
    • ‏2011-08-23T21:35:21Z
    > We have a requirement where zip files will be copied into datapower and we have extract the files and copy each file. Is it possible to do this in DP?
    >
    Yes.

    > Tried to using the zip file you have provided, but how can we extract the file from attachment?
    >
    See below template code snippet.

    It extracts file "filename-you-want-to-extract" from the zip file which is attached under Content-ID "backup.zip" of a SOAP with attachment file as binaryNode.

    The step after that attaches the extracted (Non-XML) file as new attachment (just as a sample).
    You could work with the extrcated binary file in XSLT also by accessing it like
    "dp:binary-encode(tmp/result/binary/child::node())"

    <pre class="jive-pre"> ... <xsl:variable name= "filename" select= "'filename-you-want-to.extract')"/> <xsl:variable name= "tmp"> <dp:url-open target= "{concat('cid:backup.zip?Archive=zip&amp;Filename=',$filename)}" response= "binaryNode" /> </xsl:variable> <dp:url-open target= "{concat('attachment://out/cid:',$filename)}" response= "ignore" resolve-mode= "swa"> <xsl:copy-of select= "$tmp/result/binary/child::node()"/> </dp:url-open> ... </pre>
    Hermann<myXsltBlog/>
    Thanks for the response.

    I tried using a binary transformation after the second results action and what I have noticed is that attachements was deleted after the second results action.

    And also the {concat('cid:backup.zip?Archive=zip&Filename=',$filename)}" will be converted to cid:///backup.zip?Archive=zip&Filename=file.xml?

    Is this right?...
  • HermannSW
    HermannSW
    4723 Posts

    Re: validate multiple xml in a zip

    ‏2011-08-24T13:21:47Z  
    Thanks for the response.

    I tried using a binary transformation after the second results action and what I have noticed is that attachements was deleted after the second results action.

    And also the {concat('cid:backup.zip?Archive=zip&Filename=',$filename)}" will be converted to cid:///backup.zip?Archive=zip&Filename=file.xml?

    Is this right?...
    >
    > I tried using a binary transformation after the second results action and what I have noticed is that attachements was deleted after the second results action.
    >
    not sure what happened.

    > And also the {concat('cid:backup.zip?Archive=zip&Filename=',$filename)}" will be converted to cid:///backup.zip?Archive=zip&Filename=file.xml?
    >
    > Is this right?...
    >
    That is the syntax required, "cid:///backup.zip?Archive=zip&Filename=file.xml" references
    attachment with Content-ID "backup.zip", tells that it is a zip archive (DataPower supports "tar", too),
    and references the file to extract.

    Find working examples of how to extract a

     
    Hermann<myXsltBlog/>
  • dpdeveloper
    dpdeveloper
    111 Posts

    Re: validate multiple xml in a zip

    ‏2011-08-24T22:26:45Z  
    • HermannSW
    • ‏2011-08-24T13:21:47Z
    >
    > I tried using a binary transformation after the second results action and what I have noticed is that attachements was deleted after the second results action.
    >
    not sure what happened.

    > And also the {concat('cid:backup.zip?Archive=zip&Filename=',$filename)}" will be converted to cid:///backup.zip?Archive=zip&Filename=file.xml?
    >
    > Is this right?...
    >
    That is the syntax required, "cid:///backup.zip?Archive=zip&Filename=file.xml" references
    attachment with Content-ID "backup.zip", tells that it is a zip archive (DataPower supports "tar", too),
    and references the file to extract.

    Find working examples of how to extract a

     
    Hermann<myXsltBlog/>
    Thanks Hermann.

    I will updated after testing this.