Topic
11 replies Latest Post - ‏2011-08-24T22:26:45Z by dpdeveloper
bionicl
bionicl
13 Posts
ACCEPTED ANSWER

Pinned topic validate multiple xml in a zip

‏2010-11-15T07:55:35Z |
Hi all,

im seeking advice with regards to the following scenario.

1)client application will do a batch processing and zip up multiple xml into 1 zip file (call abc.zip) and place it into a specific directory
2)dp will do a ftp poll on that directory and retrieve the abc.zip file
3)the use of dp here will be to validate each individual xml in the zip file for xml threats (no schema validation is required)
4)upon success/completion, dp will rename the zip file as abc_processed.zip and place into another directory.

i'm wondering how to go about implementing this? as in does dp has such capability to do so?

thanks and sorry for the trouble!

:)
Updated on 2011-08-24T22:26:45Z at 2011-08-24T22:26:45Z by dpdeveloper
  • mmatamoros
    mmatamoros
    60 Posts
    ACCEPTED ANSWER

    Re: validate multiple xml in a zip

    ‏2010-11-17T17:45:06Z  in response to bionicl
    what you can do is the following:

    • After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
    • The you do a dp:parse to get the XML files from the base64 string
    • Do the validations
    • Then process back the zip file with deflate and send it to the backend
    • HermannSW
      HermannSW
      4379 Posts
      ACCEPTED ANSWER

      Re: validate multiple xml in a zip

      ‏2010-11-18T17:39:20Z  in response to mmatamoros
      > what you can do is the following:
      >
      > - After you fetch the zip file do an inflate where the result is a Base64 repesentation of the XML files contained in the zip.
      > - The you do a dp:parse to get the XML files from the base64 string
      > - Do the validations
      > - Then process back the zip file with deflate and send it to the backend

      .zip is a compressed archive format, not sure how you can inflate that ...
      What I would propose instead is the following two step approach:
      1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
      https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701

      2) extract the files from that attachment (the .zip file) using "var://local/attachment-manifest",
      see chapter "Attachment protocol" from XSL Developers Accelerator guide:
      http://www-01.ibm.com/software/integration/datapower/library/documentation/v3.8.0/XS40.html

      Hermann.
      • mmatamoros
        mmatamoros
        60 Posts
        ACCEPTED ANSWER

        Re: validate multiple xml in a zip

        ‏2010-11-18T17:44:56Z  in response to HermannSW
        As per the Extension Functions Catalog the extension inflate() says:

        Decompresses a Base64-encoded string and returns a decoded version of the string.

        dp:inflate(text, algorithm)

        Algorithm

        inflate
        (Default) Decompresses the string with the algorithm that is
        described in RFC 1951. The format is LZ77 with Huffman encoding
        (Deutsch with an Adler checksum).

        gzip Decompresses the string with the algorithm that is described in
        RFC 1952. The format is LZ77 with a 32-bit checksum. (Deutsch
        without an Adler checksum).
        compress

        Decompresses the string with the algorithm that is described in
        RFC 1950. The format is adaptive LZW.
        • HermannSW
          HermannSW
          4379 Posts
          ACCEPTED ANSWER

          Re: validate multiple xml in a zip

          ‏2010-11-18T19:31:48Z  in response to mmatamoros
          > As per the Extension Functions Catalog the extension inflate() says:
          >
          > Decompresses a Base64-encoded string and returns a decoded version of the string.
          >
          > dp:inflate(text, algorithm)
          >

          Try it and you will see the problem.
          dp:inflate() returns the decoded "string" which has been compressed and then base64 encoded before.

          But a .zip file is an "archive" format.
          A .zip file typically contains more than one file.
          Not sure what dp:inflate() would give you, definitely only one "string", but how do you want to extract the contained files from that?
          From DataPower XSL Accelerator Developers Guide:

          Attachment protocol
          The fetch, results, and results-async actions support the attachment protocol.
          This protocol identifies a SOAP attachment and has the following format:
          attachment://context/cid:content_id[?query_param_1&query_param_2]

          The fetch action supports the following query parameters:
          • Encode=base64
          • Compress=gzip
          • Archive=tar or Archive=zip
          • Filename=file_name
          The results and results-async actions support the following query parameters:
          • Decode=base64 or Decode=hexbin
          • Compress=gzip
          • Archive=tar or Archive=zip
          • Filename=file_name
          • Parsable=true
          • Manifest=true

          If you want to get an idea what you can do with the attachment protocol you might want see this slide:
          http://www-01.ibm.com/support/docview.wss?uid=swg27019119&aid=1#page=8

          Hermann.
      • HermannSW
        HermannSW
        4379 Posts
        ACCEPTED ANSWER

        Re: validate multiple xml in a zip

        ‏2010-11-18T20:16:56Z  in response to HermannSW
        > ...
        > What I would propose instead is the following two step approach:
        > 1) have a non-XML request type and attach the .zip file as in this posting of Jaime:
        > https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14086701&#14086701
        > ...

        I found an even better posting of Steven Loscalpio back from 2007:
        "Re: Creating a SOAP attachment from binary file polled from FTP"
        https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14140244&#14140244

        The posting subject matches really good what bionici asked for above:
        > 2) dp will do a ftp poll on that directory and retrieve the abc.zip file
        Find attached a 3.7.3 export of XML FW makeSwa listening on port 9999.

        Here you can see it in action:
        
        $ cat testdata abtdaa $ od -Ax -tx1 testdata 000000 61 62 03 74 64 61 01 61 0a 000009 $ curl -d @testdata http:
        //dp1-l3.boeblingen.de.ibm.com:9999     --1d72d876-84e0-4877-a946-7a76566d083e Content-Type: text/xml; charset=UTF-8 Content-Transfer-Encoding: binary   <?xml version=
        "1.0" encoding=
        "UTF-8"?> <env:Envelope xmlns:dp=
        "http://www.datapower.com/extensions" xmlns:env=
        "http://schemas.xmlsoap.org/soap/envelope/"><env:Body> WhateverYouWant </env:Body></env:Envelope> --1d72d876-84e0-4877-a946-7a76566d083e Content-ID: <backup.zip> Content-Transfer-Encoding: binary Content-Type: application/x-www-form-urlencoded   abtdaa --1d72d876-84e0-4877-a946-7a76566d083e-- $ $ cat bin.xsl <?xml version=
        "1.0" encoding=
        "utf-8"?> <xsl:stylesheet xmlns:xsl=
        "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
        "http://www.datapower.com/extensions" version=
        "1.0">   <dp:input-mapping href=
        "bin.ffd" type=
        "ffd"/>   <xsl:output method=
        "xml"/>   <xsl:template match=
        "/"> <env:Envelope xmlns:env=
        "http://schemas.xmlsoap.org/soap/envelope/"> <env:Body> WhateverYouWant </env:Body> </env:Envelope> </xsl:template>   </xsl:stylesheet> $ $ cat bin.ffd <?xml version=
        "1.0" encoding=
        "utf-8"?> <File name=
        "file"> <Field name=
        "binaryNode" type=
        "binaryNode"/> </File> $
        


        Hermann.

        Attachments

        • bionicl
          bionicl
          13 Posts
          ACCEPTED ANSWER

          Re: validate multiple xml in a zip

          ‏2010-11-21T14:52:48Z  in response to HermannSW
          alright guys. thanks alot for all your responses!

          i will give it a try. keep you guys updated ;)
      • dpdeveloper
        dpdeveloper
        111 Posts
        ACCEPTED ANSWER

        Re: validate multiple xml in a zip

        ‏2011-08-23T21:21:52Z  in response to HermannSW
        We have a requirement where zip files will be copied into datapower and we have extract the files and copy each file. Is it possible to do this in DP?

        Tried to using the zip file you have provided, but how can we extract the file from attachment?
        • HermannSW
          HermannSW
          4379 Posts
          ACCEPTED ANSWER

          Re: validate multiple xml in a zip

          ‏2011-08-23T21:35:21Z  in response to dpdeveloper
          > We have a requirement where zip files will be copied into datapower and we have extract the files and copy each file. Is it possible to do this in DP?
          >
          Yes.

          > Tried to using the zip file you have provided, but how can we extract the file from attachment?
          >
          See below template code snippet.

          It extracts file "filename-you-want-to-extract" from the zip file which is attached under Content-ID "backup.zip" of a SOAP with attachment file as binaryNode.

          The step after that attaches the extracted (Non-XML) file as new attachment (just as a sample).
          You could work with the extrcated binary file in XSLT also by accessing it like
          "dp:binary-encode(tmp/result/binary/child::node())"

          
          ... <xsl:variable name=
          "filename" select=
          "'filename-you-want-to.extract')"/>   <xsl:variable name=
          "tmp"> <dp:url-open target=
          "{concat('cid:backup.zip?Archive=zip&amp;Filename=',$filename)}" response=
          "binaryNode" /> </xsl:variable>   <dp:url-open target=
          "{concat('attachment://out/cid:',$filename)}" response=
          "ignore" resolve-mode=
          "swa"> <xsl:copy-of select=
          "$tmp/result/binary/child::node()"/> </dp:url-open> ...
          

          Hermann<myXsltBlog/>
          • dpdeveloper
            dpdeveloper
            111 Posts
            ACCEPTED ANSWER

            Re: validate multiple xml in a zip

            ‏2011-08-23T22:27:41Z  in response to HermannSW
            Thanks for the response.

            I tried using a binary transformation after the second results action and what I have noticed is that attachements was deleted after the second results action.

            And also the {concat('cid:backup.zip?Archive=zip&Filename=',$filename)}" will be converted to cid:///backup.zip?Archive=zip&Filename=file.xml?

            Is this right?...