Topic
  • 11 replies
  • Latest Post - ‏2016-01-13T22:08:11Z by HermannSW
HermannSW
HermannSW
6387 Posts

Pinned topic Processing binary data returned by dp:url-open()

‏2011-02-23T14:43:00Z |
Some time ago we had a "snowman" contest.

People built and took photos of snowmans they created.

I was the only submitter of a virtual snowman -- an ASCII art picture of a snowman created by DataPower.

Below stylesheet does
  • pass the URL of a JPG snowman picture to a converter website by dp:url-open() call
  • takes the Non-XML (HTML) response and converts it to a hex string by method described in [1]
  • replaces all dots by " "s
  • converts the modified hex string back to a binary node by dp:radix-convert(_,16,64) and dp:binary-decode()
  • returns this Non-XML data by the method described in [1]
  • sets the Content-Type of the response to "text/html".

Now create a simple
  • loopback XML FW
  • with Non-XML request tyoe
  • a match action acception all
  • listening on port 3333
  • with Transform binary action using stylesheet snowman.xsl

Then open "http://yourbox:3333" in a web browser and you will see a modified ASCII art snowman in your browser.

While this special application is just fun, you can see all the aspects of handling binary data within a DataPower stylesheet in action.

See stylesheet "snowman.xsl" for the details (also attached):

<!-- snowman.xsl --> <xsl:stylesheet version=
"1.0" xmlns:xsl=
"http://www.w3.org/1999/XSL/Transform" xmlns:dp=
"http://www.datapower.com/extensions" xmlns:regexp=
"http://exslt.org/regular-expressions" extension-element-prefixes=
"dp" > <!-- 

for binary data output --> <dp:output-mapping  href=
"store:///pkcs7-convert-input.ffd" type=
"ffd"/>   <!-- b&w snowman 
"http://www.picturesof.net/_images_300/Black_and_White_Cartoon_Snowman_with_a_Tophat_Striped_Scarf_Carrot_Nose_and_a_Broom_Royalty_Free_Clipart_Picture_101113-042053-590053.jpg" --> <xsl:variable name=
"image" select=
"'http%3A%2F%2Fwww.picturesof.net%2F_images_300%2FBlack_and_White_Cartoon_Snowman_with_a_Tophat_Striped_Scarf_Carrot_Nose_and_a_Broom_Royalty_Free_Clipart_Picture_101113-042053-590053.jpg'"/>   <!-- ASCII Art Generator on 
"http://www.glassgiant.com/ascii/" --> <xsl:variable name=
"service1" select=
"'http://www.glassgiant.com/ascii/ascii.php?maxwidth=100&amp;fontsize=8&amp;webaddress='"/> <xsl:variable name=
"service2" select=
"'&amp;ggfile=&amp;MAX_FILE_SIZE=3145728&amp;negative=N'"/>     <xsl:template match=
"/">   <!-- get the HTML response from glassgiant.com --> <xsl:variable name=
"html"> <dp:url-open target=
"{concat($service1,$image,$service2)}" response=
"binaryNode"/> </xsl:variable>   <!-- convert to hex --> <xsl:variable name=
"hex" select=
"dp:radix-convert(dp:binary-encode($html/result/binary),64,16)"/>   <!-- 
"." --> <xsl:variable name=
"from" select=
"'2E'" />   <!-- 
" " --> <xsl:variable name=
"to" select=
"'266E6273703B'" />   <!-- replace all dots by   --> <xsl:variable name=
"final" select=
"regexp:replace($hex,$from,'g',$to)" />   <!-- 

for dp:output-mapping --> <object> <message> <xsl:copy-of select=
"dp:binary-decode(dp:radix-convert($final,16,64))"/> </message> </object> <!-- set content type 

for correct display of binary transform result in browser --> <dp:set-response-header name=
"'Content-Type'" value=
"'text/html'"/>   </xsl:template> </xsl:stylesheet>


[1] "DataPower "binary" node type (binaryNode)"
https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14572518&#14572518

Hermann.
Updated on 2012-05-15T00:05:31Z at 2012-05-15T00:05:31Z by HermannSW
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2011-02-23T14:44:50Z  
    And if you want to get an idea of what gets created withouit creating the DataPower snowman service yourself ...

    ... just open attached snowman.html

    Hermann.
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2011-02-23T22:26:42Z  
    • HermannSW
    • ‏2011-02-23T14:44:50Z
    And if you want to get an idea of what gets created withouit creating the DataPower snowman service yourself ...

    ... just open attached snowman.html

    Hermann.
    Here seems to be a good place to show how to deal with "real" binary data (JPEG, not just HTML)
    without further modification as response to dp:url-open().

    Stylesheet snowman0.xsl (also attached) just
    • downloads the b&w snowman JPEG with dp:url-open()
    • copies the binary node into /object/message for binary output
    • sets content-type to image/jpeg for correct display in browser
    Just put this stylesheet into the loopback XML FW described above and you will see the jpeg image for http://yourbox:3333.

    snowman0.xsl:
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <!-- 
    
    for binary data output --> <dp:output-mapping  href=
    "store:///pkcs7-convert-input.ffd" type=
    "ffd"/>   <!-- b&w snowman  --> <xsl:variable name=
    "image" select=
    "'http://www.picturesof.net/_images_300/Black_and_White_Cartoon_Snowman_with_a_Tophat_Striped_Scarf_Carrot_Nose_and_a_Broom_Royalty_Free_Clipart_Picture_101113-042053-590053.jpg'"/>   <xsl:template match=
    "/">   <!-- get the JPEG --> <xsl:variable name=
    "jpeg"> <dp:url-open target=
    "{$image}" response=
    "binaryNode"/> </xsl:variable>   <!-- 
    
    for dp:output-mapping --> <object> <message> <xsl:copy-of select=
    "$jpeg/result/binary/child::node()"/> </message> </object> <!-- set content type 
    
    for correct display of binary transform result in browser --> <dp:set-response-header name=
    "'Content-Type'" value=
    "'image/jpeg'"/>   </xsl:template> </xsl:stylesheet>
    


    Hermann.
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2011-02-23T22:35:29Z  
    • HermannSW
    • ‏2011-02-23T22:26:42Z
    Here seems to be a good place to show how to deal with "real" binary data (JPEG, not just HTML)
    without further modification as response to dp:url-open().

    Stylesheet snowman0.xsl (also attached) just
    • downloads the b&w snowman JPEG with dp:url-open()
    • copies the binary node into /object/message for binary output
    • sets content-type to image/jpeg for correct display in browser
    Just put this stylesheet into the loopback XML FW described above and you will see the jpeg image for http://yourbox:3333.

    snowman0.xsl:
    <pre class="jive-pre"> <xsl:stylesheet version= "1.0" xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" extension-element-prefixes= "dp" > <!-- for binary data output --> <dp:output-mapping href= "store:///pkcs7-convert-input.ffd" type= "ffd"/> <!-- b&w snowman --> <xsl:variable name= "image" select= "'http://www.picturesof.net/_images_300/Black_and_White_Cartoon_Snowman_with_a_Tophat_Striped_Scarf_Carrot_Nose_and_a_Broom_Royalty_Free_Clipart_Picture_101113-042053-590053.jpg'"/> <xsl:template match= "/"> <!-- get the JPEG --> <xsl:variable name= "jpeg"> <dp:url-open target= "{$image}" response= "binaryNode"/> </xsl:variable> <!-- for dp:output-mapping --> <object> <message> <xsl:copy-of select= "$jpeg/result/binary/child::node()"/> </message> </object> <!-- set content type for correct display of binary transform result in browser --> <dp:set-response-header name= "'Content-Type'" value= "'image/jpeg'"/> </xsl:template> </xsl:stylesheet> </pre>

    Hermann.
    hmmm -- attaching did not work, once again ...

    Hermann.
  • msmps
    msmps
    212 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2012-05-14T14:56:51Z  
    • HermannSW
    • ‏2011-02-23T22:35:29Z
    hmmm -- attaching did not work, once again ...

    Hermann.
    thanks
    I used the same xsl. but in my case the output is a token. And it is dispalying output but I need it in xml tags. How to do that
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2012-05-14T16:22:07Z  
    • msmps
    • ‏2012-05-14T14:56:51Z
    thanks
    I used the same xsl. but in my case the output is a token. And it is dispalying output but I need it in xml tags. How to do that
    Hi Robert,

    > thanks
    > I used the same xsl. but in my case the output is a token. And it is dispalying output but I need it in xml tags. How to do that

    This gets token from 'http://stamm-wilbrandt.de/en/xsl-list/msmps.txt' and outputs it with tag:
    
    $ echo 
    "<foobar/>" | coproc2 get-token.xsl - http:
    //dp6-l3:2223; echo <token>token123</token> $
    


    get-token.xsl (also attached):
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <xsl:output omit-xml-declaration=
    "yes"/>   <xsl:variable name=
    "url" select=
    "'http://stamm-wilbrandt.de/en/xsl-list/msmps.txt'"/> <xsl:template match=
    "/"> <!-- get token --> <xsl:variable name=
    "resp"> <dp:url-open target=
    "{$url}" response=
    "binaryNode"/> </xsl:variable>   <xsl:variable name=
    "resp64" select=
    "dp:binary-encode($resp/result/binary/child::node())"/>   <token> <xsl:value-of select=
    "dp:decode($resp64,'base-64')"/> </token> </xsl:template> </xsl:stylesheet>
    


     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • msmps
    msmps
    212 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2012-05-14T18:36:07Z  
    • HermannSW
    • ‏2012-05-14T16:22:07Z
    Hi Robert,

    > thanks
    > I used the same xsl. but in my case the output is a token. And it is dispalying output but I need it in xml tags. How to do that

    This gets token from 'http://stamm-wilbrandt.de/en/xsl-list/msmps.txt' and outputs it with tag:
    <pre class="jive-pre"> $ echo "<foobar/>" | coproc2 get-token.xsl - http: //dp6-l3:2223; echo <token>token123</token> $ </pre>

    get-token.xsl (also attached):
    <pre class="jive-pre"> <xsl:stylesheet version= "1.0" xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" extension-element-prefixes= "dp" > <xsl:output omit-xml-declaration= "yes"/> <xsl:variable name= "url" select= "'http://stamm-wilbrandt.de/en/xsl-list/msmps.txt'"/> <xsl:template match= "/"> <!-- get token --> <xsl:variable name= "resp"> <dp:url-open target= "{$url}" response= "binaryNode"/> </xsl:variable> <xsl:variable name= "resp64" select= "dp:binary-encode($resp/result/binary/child::node())"/> <token> <xsl:value-of select= "dp:decode($resp64,'base-64')"/> </token> </xsl:template> </xsl:stylesheet> </pre>

     
    Hermann<myXsltBlog/> <myXsltTweets/>
    why can't we use dp:binary-decode at the end.
    when I use binary-decode, it gave <token>***BINARY NODE***</token>. Whats wrong with that
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2012-05-15T00:05:31Z  
    • msmps
    • ‏2012-05-14T18:36:07Z
    why can't we use dp:binary-decode at the end.
    when I use binary-decode, it gave <token>***BINARY NODE***</token>. Whats wrong with that
    > why can't we use dp:binary-decode at the end.
    >
    because that results in a binaryNode.
    The only safe operation you can do on a binaryNode is dp:binary-encode().
    Both, xsl:value-of and xsl:copy-of result in ***BINARY NODE*** by design.

    > when I use binary-decode, it gave <token>***BINARY NODE***</token>. Whats wrong with that
    >
    Even if you could access the value of the binary node it would be risky and make the box unreliable.
    Please see slides 15 and 5 of these two WSTE webcasts:
    http://www-01.ibm.com/support/docview.wss?uid=swg27022977
    http://www-01.ibm.com/support/docview.wss?uid=swg27022979

    Please see slide 16 of 1st webcast above on dp:decode(_,'base-64') and that is does UTF-8 validation.
    This is necessary because DataPower internal encoding is UTF-8, see slide 11 of 1st webcast.

    In case the "token" you receive is not UTF-8 encoded it will fail dp:decode().
    Then there other methods of accessing its value, please tell if that is the case for you.

     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • pratapV
    pratapV
    9 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2016-01-13T07:09:34Z  

    HI hermann,

    I need to expose a service through the IBM APIM. The service is a download pdf. when i invoke URL in browser it gives me a pdf download.

    My service flow goes like this..

    WAS > IIB > DP(APIM) > INTERNET

    When I am exposing It is just ginving me NULL 204 no content. (back-side connection successful )

    where it is parsed in IBM IIB v10

    Find the response from IIB in attachment.

     

     

    Please help me with Inputs.

    how could i parse this in data power

    Attachments

    Updated on 2016-01-13T07:36:55Z at 2016-01-13T07:36:55Z by pratapV
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2016-01-13T08:34:43Z  
    • pratapV
    • ‏2016-01-13T07:09:34Z

    HI hermann,

    I need to expose a service through the IBM APIM. The service is a download pdf. when i invoke URL in browser it gives me a pdf download.

    My service flow goes like this..

    WAS > IIB > DP(APIM) > INTERNET

    When I am exposing It is just ginving me NULL 204 no content. (back-side connection successful )

    where it is parsed in IBM IIB v10

    Find the response from IIB in attachment.

     

     

    Please help me with Inputs.

    how could i parse this in data power

    Hi,

    how do you access the (binary) PDF?
    As service backend response, or via <dp:url-open>?

    For <dp:url-open> you need response="binaryNode" as shown in stylehseet attached above and described in the WSTE webcast links above.


    Hermann.

    Updated on 2016-01-13T08:35:34Z at 2016-01-13T08:35:34Z by HermannSW
  • pratapV
    pratapV
    9 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2016-01-13T09:11:30Z  
    • pratapV
    • ‏2016-01-13T07:09:34Z

    HI hermann,

    I need to expose a service through the IBM APIM. The service is a download pdf. when i invoke URL in browser it gives me a pdf download.

    My service flow goes like this..

    WAS > IIB > DP(APIM) > INTERNET

    When I am exposing It is just ginving me NULL 204 no content. (back-side connection successful )

    where it is parsed in IBM IIB v10

    Find the response from IIB in attachment.

     

     

    Please help me with Inputs.

    how could i parse this in data power

    Hi Hermann,
     
    I access as Service response.
    Where IIb is receiving the XML content with Binary data. When I am Invoking the URL of IIB endpoint in Browser it is dowloading(pdf file)
    When coming to DP it is null. (response)
  • HermannSW
    HermannSW
    6387 Posts

    Re: Processing binary data returned by dp:url-open()

    ‏2016-01-13T22:08:11Z  
    • pratapV
    • ‏2016-01-13T09:11:30Z
    Hi Hermann,
     
    I access as Service response.
    Where IIb is receiving the XML content with Binary data. When I am Invoking the URL of IIB endpoint in Browser it is dowloading(pdf file)
    When coming to DP it is null. (response)

    Please take a packet capture and use Wireshark to see whether backend does return a response to DataPower or only a HTTP 204.

    In case you see content being returned (I doubt that becaus of HTTP 204), then please attach a sample config here.

    Otherwise fix the backend.


    Hermann.