• Share
  • ?
  • Profiles ▼
  • Communities ▼
  • Apps ▼

Blogs

  • My Blogs
  • Public Blogs
  • My Updates

This community can have members from outside your organization. HermannSW

  • Log in to participate

▼ Tags

▼ Similar Entries

Howto: easy callout ...

Blog: HermannSW
HermannSW 2700006U54
Updated
0 people like thisLikes 0
No CommentsComments 0

DataPower as a REST ...

Blog: Me, myself an...
Offline 0600018ME9
Updated
1 people likes thisLikes 1
No CommentsComments 0

Understanding the NF...

Blog: Application I...
Aidan Harbison 310000JE0Y
Updated
4 people like thisLikes 4
No CommentsComments 0

DataPower and HTTP P...

Blog: Application I...
Chris Sloan 27000763MF
Updated
3 people like thisLikes 3
No CommentsComments 0

Gateway Scripts and ...

Blog: DataPower-bit...
Rohit-Goyal 270005E9M2
Updated
0 people like thisLikes 0
CommentsComments 1

▼ Archive

  • March 2018
  • November 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • October 2016
  • May 2016
  • April 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • February 2012
  • January 2012
  • December 2011
  • November 2011
  • October 2011
  • September 2011
  • August 2011
  • July 2011
  • June 2011
  • May 2011
  • April 2011
  • March 2011
  • February 2011
  • January 2011

▼ Blog Authors

HermannSW

View All Entries
Clicking the button causes a full page refresh. The user could go to the "Entry list" region to view the new content.) Entry list

DataPower non-XML data processing options

HermannSW 2700006U54 | | Tags:  contivo gatewayscript dataglue wtx datapower xslt ffd proccessing xquery binary non-xml ‎ | 4 Comments ‎ | 11,637 Views

Over the years I was (and still am) interested in non-XML data processing with DataPower.
I did numerous developerWorks DataPower forum postings on that, as well as WSTE webcasts.
And in this blog now 23 postings are tagged with non-xml:
https://www.ibm.com/developerworks/community/blogs/HermannSW/tags/non-xml

This post was motivated by Ruben's question on GatewayScript binary data processing methods:
https://www.ibm.com/developerworks/community/forums/html/topic?id=55fec5ff-4b1e-449c-adbc-0c51f4b0839c#48c75129-0e7b-45ac-8807-d31a5f2618fd

The whole thread discussed that handcrafted FFDs are not supported and referred back to this 2012 posting on the options. It also listed the only Enhancement Request that has been done since 2007 for FFD processing (FFD PMRs were fixed of course).

Then I remembered my Binary data processing without DataGlue license! blog posting allowing to do binary data processing without WTX and without FFD. I really like that blog post's graphical flow diagram, have a look.

Further below I will show how easily binary data processing can be done with GatewayScript (available with 7.0.0.0 firmware). But before lets summarize all DataPower Non-XML data processing options here in one place:

  1. WTX
  2. Contivo FFDs: you need Contivo Analyst product,
    no handcrafted FFDs are supported (you cannot raise PMRs against handmade FFDs)
  3. there are 4 exceptional, simple FFDs that are supported, see the WebCasts on basics and advanced techniques
  4. "Binary data processing without DataGlue license!" technique, with
    1. ... <hexbin>...<hexbin> --(XSLT)-- base64 ...
    2. ... <hexbin>...<hexbin> --(XQuery)-- base64 ... (with 6.0.0.0 or higher firmware, allows for XPath 2.0)
  5. GatewayScript


One comment on option 4. While this works without Dataglue license (these days a XG45 without DIM option) you have to "pay" the price in form of added latency and memory consumption of the attachment processing needed by that technique.
 

GatewayScript Non-XML data processing

The simple GatewayScript data structure for processing binary data is the buffer object.

For reading binary input we use readAsBuffer() method, and its documentation tries to move people to use the Buffers object.

When contexts are small, use the readAsBuffer() function. Use the readAsBuffers() function when a context is large. The readAsBuffer() function requires a contiguous memory allocation. The readAsBuffer() function is faster but is more demanding on the memory system. The readAsBuffers() function does not require contiguous memory to populate the Buffers object.


Use of Buffers might be valid for some Non-XML processing, but when the application needs access to whole input I prefer buffer.

Good news is that the first (workable) Non-XML sample program can be found in readAsBuffer() documentation itself. It is a binary identity operation with error handling. Here you can see rAB.js:


Since binary identity is not that interesting lets see now the binary reverse operation from "... without Dataglue license" posting. Adding 5 lines to rAB.js does the job. here is reverse.js:

 

Now lets see what both do on sample input from "... without DataGlue license" posting:

$ od -tcx1 0in0put0
0000000  \0   2   b   |   ~   2   b       -  \0   t   h   a   t       i
         00  32  62  7c  7e  32  62  20  2d  00  74  68  61  74  20  69
0000020   s       t   h   e       q   u   e   s   t   i   o   n  \0
         73  20  74  68  65  20  71  75  65  73  74  69  6f  6e  00
0000037
$
$ coproc2 rAB.js 0in0put0 http://dp2-l3:2227 -s | od -tcx1
0000000  \0   2   b   |   ~   2   b       -  \0   t   h   a   t       i
         00  32  62  7c  7e  32  62  20  2d  00  74  68  61  74  20  69
0000020   s       t   h   e       q   u   e   s   t   i   o   n  \0
         73  20  74  68  65  20  71  75  65  73  74  69  6f  6e  00
0000037
$
$ coproc2 reverse.js 0in0put0 http://dp2-l3:2227 -s | od -tcx1
0000000  \0   n   o   i   t   s   e   u   q       e   h   t       s   i
         00  6e  6f  69  74  73  65  75  71  20  65  68  74  20  73  69
0000020       t   a   h   t  \0   -       b   2   ~   |   b   2  \0
         20  74  61  68  74  00  2d  20  62  32  7e  7c  62  32  00
0000037
$

 

OK, that was small input, lets process 10MB.

$ head --bytes 10M /dev/urandom > out
$ du -sb out
10485760    out
$
$ time ( coproc2 reverse.js out http://dp2-l3:2227 -s > out.2 )

real    0m1.048s
user    0m0.012s
sys    0m0.080s
$ time ( coproc2 reverse.js out.2 http://dp2-l3:2227 -s > out.3 )

real    0m1.064s
user    0m0.008s
sys    0m0.075s
$ time ( coproc2 rAB.js out.2 http://dp2-l3:2227 -s > out.4 )

real    0m0.294s
user    0m0.014s
sys    0m0.074s
$
$ diff out out.3
$ diff out.2 out.4
$
$ sha1sum out out.2
8fe128844bc9a19aac275272e243a1c4ce6adc7b  out
66590beec5c0e226ba9efc8436d119589a67e9d8  out.2
$

 


Last question to be answered is on the runtime of rAB.js and reverse.js on the 10MB input. That can be answered easily based on the ExtLatency logging target of coproc2gatewayscript again blog posting:

...,AXF=137,AGS=908,...
...,AXF=134,AGS=147,...


So the reverse operation on 10MB data (read binary data, revert, output result to context) took (908-137)=771msec.

The binary identity operation on 10MB data (read binary data, output input to context) took (147-134)=13msec.

That means that the 5 lines of diff took 758msec:

$ diff rAB.js reverse.js
6a7,11
>     for(var i=0, j=buffer.length-1; i<j; ++i, --j) {
>       var b     = buffer[i];
>       buffer[i] = buffer[j];
>       buffer[j] = b;
>     }
$

 


Btw, all I wanted and needed to know on buffer object member functions, readAsBuffer() and other stuff I did find in KnowledgeCenter.

Btw2, float and double buffer function like buf.readDoubleBE() are very powerful (reading encoded integer numbers was possible in XSLT, see slide 22 of this WSTE webcast).
 

Hermann <myBlog/> <myTweets/> | <GraphvizFiddle/> | <xqib/> | <myCE/> <myFrameless/> | 

Modified on by HermannSW 2700006U54
  • Add a Comment Add a Comment
  • Edit
  • More Actions v
  • Quarantine this Entry
Notify Other People
notification

Send Email Notification

+

Quarantine this entry

deleteEntry
duplicateEntry

Mark as Duplicate

  • Previous Entry
  • Main
  • Next Entry
Feed for Blog Entries | Feed for Blog Comments | Feed for Comments for this Entry