Topic
  • 6 replies
  • Latest Post - ‏2013-02-25T09:12:29Z by SystemAdmin
SystemAdmin
SystemAdmin
6772 Posts

Pinned topic JSON as an attachment in DataPower

‏2013-02-13T09:38:15Z |
Hello,

I have a scenario where I have to process in DataPower XI50 / XI52 appliance a request and a response with an attachment in JSON format. The attachment can be a plain JSON file or JSON file compressed with a HPACK algorithm.

Can someone help me understand if its possible to achieve in DataPower and if yes then the way forward ?

Thanks in advance.
Updated on 2013-02-25T09:12:29Z at 2013-02-25T09:12:29Z by SystemAdmin
  • HermannSW
    HermannSW
    4657 Posts

    Re: JSON as an attachment in DataPower

    ‏2013-02-14T13:27:35Z  
    Hi,

    > ... The attachment can be a plain JSON file or JSON file compressed with a HPACK algorithm.
    >
    with the old
    https://github.com/WebReflection/json.hpack/wiki
    or new
    http://webreflection.blogspot.com/2011/08/last-version-of-json-hpack.html
    HPACK?

    You say "attachment", may you please attach a sample input file to this thread?

     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: JSON as an attachment in DataPower

    ‏2013-02-18T09:26:34Z  
    • HermannSW
    • ‏2013-02-14T13:27:35Z
    Hi,

    > ... The attachment can be a plain JSON file or JSON file compressed with a HPACK algorithm.
    >
    with the old
    https://github.com/WebReflection/json.hpack/wiki
    or new
    http://webreflection.blogspot.com/2011/08/last-version-of-json-hpack.html
    HPACK?

    You say "attachment", may you please attach a sample input file to this thread?

     
    Hermann<myXsltBlog/> <myXsltTweets/>
    Thanks for the response Hermann.
    I will try to put forth the entire picture. Java based Web Services will fetch the Database records into a JSON file. We are expecting a file size of 15 MB. This is the reason we were thinking of using a HPack compression method.

    The compression is done by the Java application. In DataPower I just need to build a service (MPGW) that can process a request and/or a response as an attachment (.json / .hpk).

    1. Can you please let us know if its possible to achieve JSON as an attachment in DataPower ? If yes then any specific configuration that needs to be done in DataPower to enable this or it gets processed like any other attachment ?
    2. We are planning to gzip the compressed file further (.json -> .hpk -> .gzip), is this a good approach ?

    Many Thanks,
    Shakir
  • HermannSW
    HermannSW
    4657 Posts

    Re: JSON as an attachment in DataPower

    ‏2013-02-18T10:25:19Z  
    Thanks for the response Hermann.
    I will try to put forth the entire picture. Java based Web Services will fetch the Database records into a JSON file. We are expecting a file size of 15 MB. This is the reason we were thinking of using a HPack compression method.

    The compression is done by the Java application. In DataPower I just need to build a service (MPGW) that can process a request and/or a response as an attachment (.json / .hpk).

    1. Can you please let us know if its possible to achieve JSON as an attachment in DataPower ? If yes then any specific configuration that needs to be done in DataPower to enable this or it gets processed like any other attachment ?
    2. We are planning to gzip the compressed file further (.json -> .hpk -> .gzip), is this a good approach ?

    Many Thanks,
    Shakir
    > ...
    > 1. Can you please let us know if its possible to achieve JSON as an attachment in DataPower ?
    >
    Does that mean you will send SOAP with (JSON) Attachment file?
    Or do you send JSON and DataPower should attach?
    (DataPower can process both)

    > If yes then any specific configuration that needs to be done in DataPower to enable this or it gets processed like any other attachment ?
    >
    There is no built in support for h(un)pack.

    If you answer the question from last posting on which version you are interested in, I can look into.

    > 2. We are planning to gzip the compressed file further (.json -> .hpk -> .gzip), is this a good approach ?
    >
    You should compare ".json -> .hpk -> .gzip" with ".json -> .gzip".

     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: JSON as an attachment in DataPower

    ‏2013-02-20T05:43:59Z  
    • HermannSW
    • ‏2013-02-18T10:25:19Z
    > ...
    > 1. Can you please let us know if its possible to achieve JSON as an attachment in DataPower ?
    >
    Does that mean you will send SOAP with (JSON) Attachment file?
    Or do you send JSON and DataPower should attach?
    (DataPower can process both)

    > If yes then any specific configuration that needs to be done in DataPower to enable this or it gets processed like any other attachment ?
    >
    There is no built in support for h(un)pack.

    If you answer the question from last posting on which version you are interested in, I can look into.

    > 2. We are planning to gzip the compressed file further (.json -> .hpk -> .gzip), is this a good approach ?
    >
    You should compare ".json -> .hpk -> .gzip" with ".json -> .gzip".

     
    Hermann<myXsltBlog/> <myXsltTweets/>
    Hello Hermann,

    We will be sending SOAP with JSON attachment and DataPower has to process it in terms of validating the attachment to be correct and not corrupted, and then pass it to the back end system.

    I am looking more interested in the newer version of HPack algorithm and if that works it would be great.

    Thanks.

    Regards,
    Shakir
  • HermannSW
    HermannSW
    4657 Posts

    Re: JSON as an attachment in DataPower

    ‏2013-02-20T11:54:05Z  
    Hello Hermann,

    We will be sending SOAP with JSON attachment and DataPower has to process it in terms of validating the attachment to be correct and not corrupted, and then pass it to the back end system.

    I am looking more interested in the newer version of HPack algorithm and if that works it would be great.

    Thanks.

    Regards,
    Shakir
    Hi,
    >
    > We will be sending SOAP with JSON attachment and DataPower has to process it in terms of validating the attachment to be correct and not corrupted, and then pass it to the back end system.
    >
    not sure why you would need to hunpack the JSON for validation, but anyway below is a stylesheet doing that.

    > I am looking more interested in the newer version of HPack algorithm and if that works it would be great.
    >
    OK, I took three samples from JSONH page.

    In order to not reinvent the wheel I made use of DataPower "JSON" request type and convert-http action to convert JSON to JSONX.
    Then stylesheet hunpack.xsl does the unpacking (with some error handling).
    The result is the "unpacked" JSONX file.
    Finally stylesheet "store:///jsonx2json.xsl" is used to generate the unpacked JSON output.

    The attached hunpack.zip contains stylesheet hunpack.xsl, the three sample files as well as all generated files.
    
    $ cat 0.json [0] $ cat 1.json [1,
    "a",
    "A",
    "B"] $ cat 2.json [2,
    "latitude",
    "longitude",1.23,5.67,2.23,6.67] $ $ curl --data-binary @0.json http:
    //dp3-l3:2052 -s > 0.jsonx $ curl --data-binary @1.json http:
    //dp3-l3:2052 -s > 1.jsonx $ curl --data-binary @2.json http:
    //dp3-l3:2052 -s > 2.jsonx $ $ coproc2 hunpack.xsl 0.jsonx http:
    //dp3-l3:2223 -s > 0.xml $ coproc2 hunpack.xsl 1.jsonx http:
    //dp3-l3:2223 -s > 1.xml $ coproc2 hunpack.xsl 2.jsonx http:
    //dp3-l3:2223 -s > 2.xml $ $ coproc2 jsonx2json.xsl 0.xml http:
    //dp3-l3:2223 -s ; echo [  ] $ coproc2 jsonx2json.xsl 1.xml http:
    //dp3-l3:2223 -s ; echo [ 
    { 
    "a":
    "A" 
    }, 
    { 
    "a":
    "B" 
    } ] $ coproc2 jsonx2json.xsl 2.xml http:
    //dp3-l3:2223 -s ; echo [ 
    { 
    "latitude":
    "1.23", 
    "longitude":
    "5.67" 
    }, 
    { 
    "latitude":
    "2.23", 
    "longitude":
    "6.67" 
    } ] $
    


    So now lets see how you can get the JSON for transformation to JSONX from SOAP with Attachment (SwA).
    Here is a sample SwA input file:
    
    $ cat 2.swa   --someUniqueBoundary Content-Type: application/soap+xml   <?xml version=
    "1.0" encoding=
    "UTF-8"?> <env:Envelope xmlns:env=
    "http://schemas.xmlsoap.org/soap/envelope/"> <env:Body> <dummySOAPnode/> </env:Body> </env:Envelope> --someUniqueBoundary Content-Type: application/json Content-ID: <json1>   [2,
    "latitude",
    "longitude",1.23,5.67,2.23,6.67] --someUniqueBoundary-- $
    


    You can send the file to DataPower any way you like (with the correct HTTP headers).
    I use swacurl tool, described in this posting:
    https://www.ibm.com/developerworks/forums/message.jspa?messageID=14657430

    I use a simple XML FW service
    • listening on port 20510
    • with request type XML
    • with "Request attachment processing mode" set to "Allow"
    • match all action
    • Fetch action, fetching "cid:" of value "json1", with Output Type "Binary"(!) on advanced tab
    • finally strip attachments action

    As you can see this simple service extracts the JSON attachment content (you can do similar with <dp:url-open ...>):
    
    $ swacurl 2.swa http:
    //dp3-l3:20510 ; echo [2,
    "latitude",
    "longitude",1.23,5.67,2.23,6.67] $
    


    This JSON output has to be sent to another service with JSON request type to do the JSON-to-JSONX conversion.
    You can do so by a chained service (on same or other device), or again, by <dp:url-open ... />

     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: JSON as an attachment in DataPower

    ‏2013-02-25T09:12:29Z  
    • HermannSW
    • ‏2013-02-20T11:54:05Z
    Hi,
    >
    > We will be sending SOAP with JSON attachment and DataPower has to process it in terms of validating the attachment to be correct and not corrupted, and then pass it to the back end system.
    >
    not sure why you would need to hunpack the JSON for validation, but anyway below is a stylesheet doing that.

    > I am looking more interested in the newer version of HPack algorithm and if that works it would be great.
    >
    OK, I took three samples from JSONH page.

    In order to not reinvent the wheel I made use of DataPower "JSON" request type and convert-http action to convert JSON to JSONX.
    Then stylesheet hunpack.xsl does the unpacking (with some error handling).
    The result is the "unpacked" JSONX file.
    Finally stylesheet "store:///jsonx2json.xsl" is used to generate the unpacked JSON output.

    The attached hunpack.zip contains stylesheet hunpack.xsl, the three sample files as well as all generated files.
    <pre class="jive-pre"> $ cat 0.json [0] $ cat 1.json [1, "a", "A", "B"] $ cat 2.json [2, "latitude", "longitude",1.23,5.67,2.23,6.67] $ $ curl --data-binary @0.json http: //dp3-l3:2052 -s > 0.jsonx $ curl --data-binary @1.json http: //dp3-l3:2052 -s > 1.jsonx $ curl --data-binary @2.json http: //dp3-l3:2052 -s > 2.jsonx $ $ coproc2 hunpack.xsl 0.jsonx http: //dp3-l3:2223 -s > 0.xml $ coproc2 hunpack.xsl 1.jsonx http: //dp3-l3:2223 -s > 1.xml $ coproc2 hunpack.xsl 2.jsonx http: //dp3-l3:2223 -s > 2.xml $ $ coproc2 jsonx2json.xsl 0.xml http: //dp3-l3:2223 -s ; echo [ ] $ coproc2 jsonx2json.xsl 1.xml http: //dp3-l3:2223 -s ; echo [ { "a": "A" }, { "a": "B" } ] $ coproc2 jsonx2json.xsl 2.xml http: //dp3-l3:2223 -s ; echo [ { "latitude": "1.23", "longitude": "5.67" }, { "latitude": "2.23", "longitude": "6.67" } ] $ </pre>

    So now lets see how you can get the JSON for transformation to JSONX from SOAP with Attachment (SwA).
    Here is a sample SwA input file:
    <pre class="jive-pre"> $ cat 2.swa --someUniqueBoundary Content-Type: application/soap+xml <?xml version= "1.0" encoding= "UTF-8"?> <env:Envelope xmlns:env= "http://schemas.xmlsoap.org/soap/envelope/"> <env:Body> <dummySOAPnode/> </env:Body> </env:Envelope> --someUniqueBoundary Content-Type: application/json Content-ID: <json1> [2, "latitude", "longitude",1.23,5.67,2.23,6.67] --someUniqueBoundary-- $ </pre>

    You can send the file to DataPower any way you like (with the correct HTTP headers).
    I use swacurl tool, described in this posting:
    https://www.ibm.com/developerworks/forums/message.jspa?messageID=14657430

    I use a simple XML FW service
    • listening on port 20510
    • with request type XML
    • with "Request attachment processing mode" set to "Allow"
    • match all action
    • Fetch action, fetching "cid:" of value "json1", with Output Type "Binary"(!) on advanced tab
    • finally strip attachments action

    As you can see this simple service extracts the JSON attachment content (you can do similar with <dp:url-open ...>):
    <pre class="jive-pre"> $ swacurl 2.swa http: //dp3-l3:20510 ; echo [2, "latitude", "longitude",1.23,5.67,2.23,6.67] $ </pre>

    This JSON output has to be sent to another service with JSON request type to do the JSON-to-JSONX conversion.
    You can do so by a chained service (on same or other device), or again, by <dp:url-open ... />

     
    Hermann<myXsltBlog/> <myXsltTweets/>
    Thanks Hermann.