HermannSW 2700006U54 Tags:  pbmtobraille graphviz braille dot xslt image non-xml unicode terminal datapower netpbm 2 Comments 164 Visits
HermannSW 2700006U54 Tags:  dot netpbm terminal non-xml pbmtobraille unicode xslt datapower image graphviz braille 7 Comments 824 Visits
"ws & comments":"http://www.fileformat.info/format/pbm/egff.htm"
This week I retweeted a last April tweet I stumbled across:
This looked really interesting to me, graphviz dot tool drawing output "inside" the terminal:
I wanted to be able to do same or similar [of course store as file and view via gimp or browser is possible].
Back in 2011 I gave 2 WSTE webcasts on "Non-XML Data Processing in WebSphere DataPower SOA Appliances Stylesheets". The 2nd webcast shows on slide 28 how I did convert a bitmap image into textual output making use of the Braille Patterns.
This is the conversion for snowman.pbm, .pbm is portable bitmap format from netpbm tools:
Typically only the top 2x3 dots of 2x4 get used, as you can see above I used all 2x4.
Here are the numbering of the dots as well as the hex codes are specifed:
There are tons of converters between different formats in netpbm tools, as well as modifiers (like eg. pnmcrop).
In this blog posting I will now talk about new pbmtobraille tool I created.
You find the C source, a syntax highlighted version and sample outputs generated in pbmtobraille.zip for download.
pbmtobraille.c[.pre.html] is a small C program (97 lines without comments containing sample outputs) doing just what it says:
pbmtobraille.c.html is just the syntax highlighted version.
samples.txt[.pre.html] contains various sample output produced (shown below), which are part of pbmtobraille.c's comments too.
9x9.pbm is a really crazy parsing sample according the .pbm spec, following this statement:
This is the header section demonstrating basic use with pbmtext output, including negation of generated output, as well as the help line telling the tool's features:
This is top of tail comment section, showing graphviz output done by pbmtobraille:
And finally this is bottom section showing a bigger layout in vertical direction (layout=TB is default, Top to Bottom):
I am a German, born 1965 in Dortmund. I did learn to read the old German characters in use until the 1st half of 20th century, but not to write them.
Today my attention was brought to this again by this tweet (pointing to a nice code piece):
First I learned from Wikipedia that the black-letter characters were in use in Germany from 1150 until 1st half of 20th centurty:
And that similar letters for use in mathematical formulas are called fraktur.
It just outputs "Hallo Welt!" which is German for "Hello world!":
Next I wanted to replace all ASCII characters in that stylesheet by their black-letter counterparts.
The string of all black-letter characters was given under above "...#unicode" Wikipedia link, and XPath translate() function𝖊𝖓𝖈𝖔𝖉𝖎𝖓𝖌 did the rest:
Here you can see it work:
$ coproc2 𝖍𝖆𝖑𝖑𝖔-𝖜𝖊𝖑𝖙.𝖝𝖘𝖑 dummy.xml http://dp2-l3:2223 ; echo
𝕳𝖆𝖑𝖑𝖔 𝖂 𝖊𝖑𝖙!
HermannSW 2700006U54 Tags:  contivo dataglue gatewayscript wtx datapower xslt ffd proccessing binary xquery non-xml 883 Visits
Over the years I was (and still am) interested in non-XML data processing with DataPower.
This post was motivated by Ruben's question on GatewayScript binary data processing methods:
The whole thread discussed that handcrafted FFDs are not supported and referred back to this 2012 posting on the options. It also listed the only Enhancement Request that has been done since 2007 for FFD processing (FFD PMRs were fixed of course).
Then I remembered my Binary data processing without DataGlue license! blog posting allowing to do binary data processing without WTX and without FFD. I really like that blog post's graphical flow diagram, have a look.
Further below I will show how easily binary data processing can be done with GatewayScript (available with 18.104.22.168 firmware). But before lets summarize all DataPower Non-XML data processing options here in one place:
The simple GatewayScript data structure for processing binary data is the buffer object.
Good news is that the first (workable) Non-XML sample program can be found in readAsBuffer() documentation itself. It is a binary identity operation with error handling. Here you can see rAB.js:
Now lets see what both do on sample input from "... without DataGlue license" posting:
$ od -tcx1 0in0put0
OK, that was small input, lets process 10MB.
$ head --bytes 10M /dev/urandom > out
The binary identity operation on 10MB data (read binary data, output input to context) took (147-134)=13msec.
That means that the 5 lines of diff took 758msec:
$ diff rAB.js reverse.js
XInclude (XML Inclusions) is a big spec that never got much adoption, at least partly because dependent on XPointer spec:
A customer wanted to know why XInclude does not work on DataPower.
Simple answer is that it is not listed under "Supported standards and protocols", here for 6.0.0.x firmware:
(XSLT does have such mechanisms, <xsl:include> and <xsl:import>)
This is document.xml:
And here can you see that it does happen with xinclude.demo.xsl:
And this is xinclude.xsl implementing "Basic Inclusion" only from XInclude spec:
Some words of caution:
In response to: Sum of Pascal's triangle reciprocalsApproved additions to these Online Encyclopedia of Integer Sequences (OEIS):
HermannSW 2700006U54 Tags:  dp:xquery-transform datapower 1.0 gatewayscript xquery firmware 2.0 xpath 22.214.171.124 xslt 959 Visits
DataPower has a XSLT 1.0 processor since day 1 long ago (1999), and that provides XPath 1.0.
With 126.96.36.199 firmware DataPower shipped XQuery 1.0 processor as well, and that provides XPath 2.0.
With 188.8.131.52 firmware released last Friday we got much new exciting stuff (I blogged on GatewayScript before).
This expression just compares false() with an empty nodeset and that is true for XPath 1.0 and false for XPath 2.0.
This XPath expresion is stored here (in order to be accessible by my and your DataPower boxes):
Now below stylesheet isXPath1.xsl does evaluate above XPath expression in a singe concat statement:
As promised this shows that XPath 1.0 and XPath 2.0 happen within a single concat statement!
Here is stylesheet isXPath1.xsl:
In developerWorks DataPower forum thread Date Conversion from Gregorain to Hijri Asif asked on how to convert Date to Hijri. Wikipedia told me that Hijri is the islamic calendar. External links section showed Calendar Converter link (http://www.fourmilab.ch/documents/calendar/) and that site is pretty cool. It converts date entered in one format to many others (Gregorian, Julian day, Julian, Hebrew, Islamic, Persian, Mayan, Bahá'í, Indian Civil, French Republican, ISO-8601, Unix time(), Excel Serial Day Number).
First lets see both in action, against XSLT and GatewayScript coproc2 endpoints:
The 2nd shows a small difference of GatewayScript to previous JSON processing on DataPower. GatewayScript implements RFC7159 which allows any JSON element at top level, while previous RFC4627 required top level element to be either JSON array or JSON object:
OK, here is stylesheet gregorian_to_islamic.xsl more or less straight translated to XSLT (the needed parts only).
Since the complete work is done in the modules, gregorian_to_islamic.js is minimal:
HermannSW 2700006U54 Tags:  stylesheet status probe debug variable xslt datapower service 1,740 Visits
I am responsible for DataPower Probe since I joined DataPower (Compiler) team 7 years ago. Just looked at the 18 Probe related APARs from the last 4 years and saw that I did fix half of them.
DataPower stylesheets are able to access all status providers via "dp:variable('var://service/system/status/___')":
2) on "disable" Probe, flush service's XML Manager's Stylesheet cache
3) In order to not get fooled by another service's Probe enabled with same XML Manager, use a XML Manager specific to each service
HermannSW 2700006U54 Tags:  swa xslt datapower mimswa attachment soap rest swacurl mimetype.java json 1,777 Visits
Customer raised question on developerWorks DataPower forum on how to convert client SOAP With Attachmnt (SWA) request on DataPower to REST. Yesterday's posting specifically raised the question on how to convert SWA to JSON-WSP attachment service request format.
Find the stylesheet doing the transformation, and how (simple) the attachments itself can be processed in my response:
The response attachment .zip contains sample service export, sample files and quite some useful tools for dealing with SWA files and DataPower. Besides the swa2json-wsp.xsl stylesheet these tools are useful for dealing with SWA files:
With these tools sending some attachments to DataPower SWA service is just two commands (see posting above for sample executions):
End of 2012 I did blog posting BMP.xsl.html (Basic Multilingual Plane) in which I described stylesheet BMP.xsl that created nice 16x16 table with each cell containing a 16x16 table containing a single unicode character, that spans the complete Basic Multilangual Plane (BMP, Unicodes 0-65535).
In this xml-dev posting John Cowan pointed to two BMP videos and an A0 .pdf poster.
I replied to that posting and criticized that the A0 poster does not show any BMP structure.
I wanted to have BMP.xq.html A0 poster, but the biggest paper format our printers do provide is A3.
Since today his A0 = 8 x A3 BMP poster covers the outside wall of my Boeblingen lab office ;-)
Recently I got a stylesheet with really HUGE number of arguments in a concat() function.
My first approach of determining the arity was not nice and needed handwork, but did give the result I needed.
Later I learned from our XQuery team that XPath 2 has a much simpler solution for that:
Just replace concat(...) by count( (...) ) making the argument list an XPath 2 sequence and applying the count() function.
With v6.0 firmware DataPower has an XQuery 1.0 compiler, so you can evaluate with DataPower.
The result is "12", which you can see live evaluated on try.zorba.io (I do like the "Share" button):
In developerWorks DataPower forum thread Need Help With XML to Json Transformation the question was raised on how to transform XML to JSON.
I did propose a transformation from XML to JSONX and then making use of "store:///jsonx2json.xsl" transformation to get the JSON output:
Pros of that approach:
I created "all-in-one" stylesheet for doing the XML-->JSON transformation rdp87.3.xsl:
Now I created two XML FW on a box with 184.108.40.206 firmware containing the (performance) fix for APAR IC90781 from May 2013 fixpack.
Finally I did:
Yes, the total time taken for rdp87.2.xsl (1961ms) in the templates is by a factor of 2.66 greater than that of rdp87.3.xsl (735ms).
But that does only make an effect if the whole operation of a service is just XML->JSON conversion.
If this is the case, then the (small) runtime overhead added is outweighted by maintainability and ease of stylesheet in my eyes.
This is the really small diff:
When using DataPower Probe sometimes a context is displayed as empty. although you know it is not.
Reason is that in DataPower Probe
All screenshots below are done for Firefox web broser, with Chrome browser it is similar.
OK, so lets take this very simple Transform stylesheet value.xsl:
So for input document "<a>1<b>2</b>3</a>" it returns just "123", see sample "curl" request and response returned:
With Probe enabled for the service we get this INPUT context ...
And the context information now is available in 0.xml.gz, 1.xml.gz, (2.xml.gz, ... if there are more contexts):
For the sample we have this:
As you can see the "real content" (123) is framed by serialized <wrap> element.
I got notice of a customer requirement to do the following with Non-XML input:
While this sounded not difficult at first, gzip(Non-XML) turned out to be not that easy to do with DataPower.
dp:deflate() cannot be used because it cannot deal with Non-XML data.
The solution was to use the Output-Filter of a rule -- that allows to compress the output data by "PKZIP" or "gzip".
So the solution presented here requires a chained service, and I am not aware of a solution to gzip Non-XML data without a chained service.
You can find a 220.127.116.11 service export here:
It provides two service endpoints, "/gz-hash" doing what is described above, and "/gz" just doing gzip(Non-XML).
Here is a short demonstration of the services:
There is a "front" XML-FW listening on port 6001, gziping the Non-XML input and dispatching
to /gz XML-FW on port 6002 or /gz-hash XML-FW on port 6003.
In order to gzip(Non-XML) both rules have to set "gzip" as Output-Filter and Non-XML Processing to "on" (via Objects screen).
The gz-hash service on port 6003 has a Non-XML Transform Action with output context "xwa" generating the hash.
Next is a Results action that attaches Non-XML input to "xwa" context. Because the Non-XML input is the gziped
input from "front" service this does the right thing.
Last a Results action returns the "xwa" context to OUTPUT (inclusive the attachment, as MIME, see above).
Here is a combination of 4 screenshots of the whole gz-hash policy.
And this is stylesheet "hash.xsl" that hashes the Non-XML input data using dp:hash-base64() DataPower extension function:
I got confirmation that the hash computed by DataPower matches the hash computed for same file by Java backend application.