I had this stylesheet lying around for some time.
It is clear that it is possible to convert a Non-XML CSV (comma seperated values) file to XML by a binary transform.
Find additional information on how to do csv2xml conversion without the need of a WTX or Contivo license in this developerWorks DataPower forum posting:
2012.xsl (431 bytes)
<update>closure in this developerWorks DataPower forum posting</update>
Ups -- out on Christmas vacation I posted this too quickly -- this posting is NOT on Stylesheet Profiling (as you might have already noticed).
This posting is on Stylesheet Debugging or Tracing.
(this previous posting was on Stylesheet Profiling: https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/entry/stylesheet_profiling_tmes8)
Unfortunately this feature got broken in 18.104.22.168 development, and therefore this feature was not available on all release
branches besides 3.7.3.x, but 3.7.3 will be EOS (end of service) in 8 days.
Today December fixpack has been published, and it contains the fix for APAR IC78912
"APPLIANCE RESTARTS WHEN USING DEBUG RULE ON XML MANAGER'S COMPILE OPTIONS POLICY FOR NON-STREAMING REQUESTS."
Attached to posting
stylesheet "des_demo.xsl" demonstrated how to do DES encryption and decryption in ECB mode.
Since ECB mode is not supported by DataPower (CBC mode is), I implemented DES completely in a stylesheet.
This is the 7.2MB HTML page of Stylesheet
des_demo.html (7.2 MB)
<EDIT date="2/5/2014" reason="broken link due to forum SW upgrade">des_demo.html (7.2MB)</EDIT>
Be careful, this file consists of 35175 lines -- it will take some time until your browser will have loaded it completely.
Looking into this HTML page you will see the very detailed output which shows the templates called, the variable settings, ...
In the middle (search for "des:encrypt-blk" with your browser) you see the output generated by the first DES encrypt operation and the start of decrypt operation.
(as you can see the encrypt operation took 45ms with generation of 17587 lines of profiling output, without stylesheet
Nice is to see the output for recursive function calls, first each line getting smaller and smaller,
and then getting longer and longer again in returning from the recursive calls.
For me this is not just
HermannSW 2700006U54 Tags:  binary datapower dp:url-open with smtp email send attachment xslt 5 Comments 11,679 Views
In this developerWorks DataPower forum posting the question was raised how to make DataPower send emails with (binary) attachments.
Technote "Sending HTML data using IBM WebSphere DataPower SOA Appliance SMTP url-open extension function" from June this year
is a first step. There we can learn that MIME="true" is necessary to do anything more than sending just a normal text email.
Next looking up InfoCenter for url-open (SMTP URLs) tells us that we have to create the raw MIME message ourselves (as in the technote above).
MIME=true | false
Stylesheet smtp.xsl (5.1 KB) just demonstrates how to do that.
Even more, it provides function mime:sendBodyWithAttachment() for easily sending an email with attachment.
This function makes use of function mime:split64() for ensuring that the base64 encoded (binary) attachment data has at most 76 characters per line (MIME spec requirement).
The attachment sent with the email is retrieved from my private website via another dp:url-open() GET request, see this slide on getgif.xsl:
Please see these two Webcasts on "Non-XML Data Processing in WebSphere DataPower SOA Appliances Stylesheets" for details on binary data processing:
Using coproc2 to execute smtp.xsl returns "sent".
The email got sent to my private email address.
This screenshot of my webmail client proves that sending the email with attachment really worked:
Last, but not least, I want to use this posting for a graphical experiment.
Since the code listings available in this blog are not as "nice" as those in developerWorks forum I took a screenshot of the
terminal with syntax highlighted listing of smtp.xsl. Since best display in this blog is for pictures of width exactly 400 pixels
I had to zoom out a little ... the resulting screenshot is a 400x1770 pixel(!) .gif image.
Good that I have a 24'' monitor in my office and a 1920x1200 graphic card for working in "Left" instead of "Normal" view ... :-)
HermannSW 2700006U54 Tags:  non-xml export backup xslt datapower attachment form-data html 1 Comment 7,807 Views
This posting describes the zip2html tool.
Sample HTML output's, service export as well as zip2html.xsl are attached in this sibling developerWorks DataPower forum thread:
zip2html tool allows to
It can be used in different modes:
See these sample outputs of zip2html tool:
backup-b4.none.attached.html (23.8 KB)
backup-b4.all.attached.html (69.3 KB)
zip2html.none.onboard.html (5.4 KB)
backup-b4.zip is a backup of the 4 domains "coproc2", "demo", "empty" (domain created, no config added) and "zip2html".
"demo" domain contains files of different types which your browser's application/zip helper should be able to correctly classify.
zip2html.zip (428.1 KB) is the XML FW service export of zip2html tool.
Add this is new export that can be used on all appliances including XB6 and XM70.
Use this URL in your web browser: http://yourBox:2228/form
zip2html.xsl (22.0 KB) is the stylesheet discussed in detail under "Technical discussion" at the bottom.
With this tool even a full device backup is processed in seconds in case "files=none".
Be careful processing big backups with option "files=all" as that might take few minutes to complete.
Just having "default" domain as part of a backup takes time in case "files=all" -- all files in "store:///" folder belong to default domain and get added.
This is how the HTML form looks like.
It allows to select the backup (or export) from your filesystem,
Then you have to select whether you want the files to be included in the generated HTML page in "data" links. This allows inspecting the files by the "application/zip" helper application registered with your browser.
And you have to select whether the attached or the onboard clixform.xsl should be used.
The clixform.xsl converts the "export.xml"s in the .zip archive to a sequence of CLI commands (Command Line Interface) during the initial phase of a DataPower import. This output is what you will see in the yellow background sections.
Find more explanations on which clixform.xsl will be used by DataPower by clicking on the screenshot or the real HTML form opened in you browser.
You open the HTML form with "/form" enpoint of your DataPower service, eg:
HTTP GET to installed serviceYou can retrieve the HTML form with correct "post" link to the DataPower box by a simple HTTP GET.
You can then use the stored form.html instead of the HTML form from DataPower box itself.
HTTP POST to installed serviceSelection of the clixform mode is done by the endpoint, the files selection is done in the query part of the URL:
Using clixform.xsl with coproc2(nonxml) serviceAs above the selection of the clixform mode is done by the endpoint and the files selection is done in the query part of the URL.
But this time stylesheet clixform.xsl is sent together with the .zip archive to the Non-XML endpoint of coproc2 service (find details here).
Different output for backups and exports
Here you can see the difference in generated output for a DataPower backup (left) and a DataPower export (right):
For backups the "domains" section contains the list of domains contained in the backup.
After each line the "go" link allows you to quickly get to the start of the domain's information.
Domain outputThis is what the domain output looks like.
Above the table you find the domain name as well as a link to the top of the document (eg. for selecting another domain by "go" link).
The table contains a configuration and a files section.
You can easily jump between both by the "files" and "configuration" links,
The "(no)" before "files" indicates that this output was generated with "files=none" selection.
For "files=all" it looks like this: "(all)"
If your browser allows you to open "data:..." links you can inspect the files by clicking on "all" link.
The "application/zip" helper application of your browser will open the base64 encoded .zip archive in the all link then.
With the description above you have everything you need to use zip2html tool.
If you are interested in some of the technical details of zip2html.xsl you are right here.
It then attaches the received backup.zip under Content-ID "sys" (cid:sys).
Next step is the extraction of export.xml from top level directory of the archive.
Then the general information HTML table at the top gets created.
Now iteration over the domains found in export.xml is done.
Each domain is contained in top level directory of received archive.
Eg. domain "dom1" will be read from cid:sys and attached as cid:dom1.zip.
The attaching allows to read export.xml from dom1.zip as well as to access
the files information for the domain by querying the attachment manifest.
In addition the files can be extracted and copied over to a newly created archive in case "files=all".
Finally the domain attachment gets removed by dp:strip-attachments before the next domain is processed.
There is a directory dp-aux in received archive (attachment cid:sys).
Above you see the inclusion hierarchy of the files in that directory.
Applying dp:transform(_,_) to clixform.xsl requires a URL as the reference to it.
Luckily DataPower attachment protocol provides exactly that URL:
"attachment://" is the protocol, "swa" is the context, "Archive=zip" says we have a zip-archive (DataPower has support for tar, too).
Finally "Filename=" points to a file in the archive, allowing to access all files at all directory levels.
The output of dp:transform() from clixform.xsl applied to export.xml is used to generate the CLI configuration output of each domain
(with yellow background color).
See the use of "localZip" if you need to create an empty archive yourself.
For each domain a new empty zip-archive gets attached to receive all files selected by "files=all" as "cid:aux".
Finally this zip-archive is read to generate the "data:application/zip;base64,..." links by just using dp:binary-decode().
At the bottom of stylesheet zip2html.xsl you find a section allowing to process "multipart/form-data" directly in the stylesheet.
Unlike swaform tool
no service chaining is required to access the fields of a posted HTML form.
swaform tool or func:filename() are needed in case you need to process binary file uploads, convert-http action cannot deal with those.
zip2html is realized by a Non-XML loopback XML firewall with "Process Messages Whose Body Is Empty" set to "on" on advanced tab.
The endpoints "/attached", "/onboard" and "/mime" are for HTTP POST requests.
Now comes a tricky part, the dp:input-mapping needed to process the POST request input data requires data on its input context.
But there is no input data for the "/form" GET request using the same stylesheet.
On the right you see the simple solution I found for this mixed GET/POST processing problem for Non-XML input processing stylesheets:
Have two rules, one for "/form" endpoint, and another for everything else.
In "/form" rule have input context NULL for "Transform Binary" action.
In the other rule have input context INPUT -- that's it.
(click on the screenshot to see all details)
HermannSW 2700006U54 Tags:  coproc2nonxml management sed datapower perl xml xslt base64 coporc2 soma 9 Comments 13,111 Views
DataPower response to a XML management do-export request is a XML file containing the export as base64 string in <dp:file>...</dp:file>.
DataPower generates the base64 string as one long string.
Some tools seem to have problems in decoding the base64 string.
Find three different decoding solutions in this developerWorks DataPower Forum posting:
* based on sed+tail+base64 commands
* a PERL one-liner
* DataPower stylesheet dec.xsl and coproc2nonxml service.
HermannSW 2700006U54 Tags:  upload convert-http action form swaform datapower file xslt http 4,428 Views
There was an interesting problem on developerWorks DataPower forum.
A "windows-1250" encoded HTTP form posted data to DataPower inclusive an uploaded file.
That file data got converted to UTF-8 because that is the DataPower internal encoding
Now that file data should be store in a database via <dp:url-open ...>, but not in UTF-8 but its original encoding.
While there is a solution for this kind of problem (slides 6-8 in this WSTE webcast)the conversion to and from UTF-8 are overhead at least.
In addition the filename of the file uploaded is not available by DataPower convert-http action.
One solution is making use of swaform tool which even is able to deal with binary file data in file upload fields (see slide 16 of this WSTE webcast).
Another solution is making use of stylesheet fileupload.xsl attached in this posting:
<EDIT>had to fix pointed to posting and add modified link here -- forum software migration broke original posting</EDIT>
You pass in the base64 encoded Non-XML data received at DataPower from the HTTP form post and the field name of the file upload field (eg. FICHIER).
Then you get back something like this:
Now you can access the filename and pass the file content unmodified by <dp:url-open target="..." data-type="base64" ...>.
In addition a convert-http action may be used to process the other HTTP form fields
(or you enhance fileupload.xsl to do that processing in addition -- no need for convert-http then).
HermannSW 2700006U54 Tags:  extension des xslt function ecb xalan datapower cbc mode 2 Comments 7,039 Views
In a developerWorks DataPower Forum thread the question was raised on how to do ECB mode DES encryption with DataPower.
The answers were two-fold:
Find the implementation with a sample from "The DES Algorithm Illustrated" in action, portable for DataPower and xalan use:
A 64bit des:encrypt-blk() call takes roughly 3ms, which means that you do not want to apply that on big data.
The implementation works of 01-strings, hexTObin and other conversion functions included.
This reminds me 5.5 years back when I worked in Smartcard development department in Boeblingen Lab.
The person responsible for side channel attacks left IBM and I was the first who raised the hand for the equipment:
What was easy to do was to break triple DES on cards without randomization counter-measure based on statistical analysis of several thousand measurements.
What I heard about is the single request break of RSA private keys in early smardcards.
The code for doing exponentiation in that cards was "efficient", it did compute
We did not have the equipment to cut reader power based on some runtime condition or at a specific clock cycle of computation.
But the certification agencies (and real attackers) had.
Once we got a complaint from a cert agency that a specific pattern on the oscilloscope would leak information.
They claimed that the pattern corresponds to a specific part of the OS code.
Using the oscilloscope I was able to disprove that statement.
I just added code that created spikes at specific locations in the OS code.
Then running the request the oscilloscope proved that the pattern of the complaint belonged to a totally different part of the OS code.
Now how to generate a "spike"?
That was easy -- just power on the (hardware) random generator and immediately power it off with the next command.
But that was long ago, before IBM sold its smartcard business and I joined DataPower development 4.5 years ago.
Yesterday's posting "Arrows in function names: func:b64⇉hex() and func:bin┈⇢dec()"
demonstrated that arrows from Unicode "Arrows" range 2190-21FF as well as other Unicode characters can be part of function names.
The attached and described stylesheet povides useful conversion functions in addition to dp:radix-convert() (see
Here a conversion of type "...
while a conversion of type "...
b64 (base64), hex, bin (01-strings) and dec (number) can be converted by these function calls:
HermannSW 2700006U54 Tags:  xpath++ webservice datapower repair tool mime broken xslt coproc2 3,722 Views
Please have a look at some interesting/useful postings from today:
Find new piggyback technique for a stylesheet executed by coproc2 service.
It allows to "call" coproc2 service from a stylesheet just being executed by coproc2 service(!):
Posting "xpath++.xsl" allows xpath++ tool to be executed on all (Java) platforms by coproc2.java client:
See sample execution here:
Finally posting "Re: MIME response with no end tag through MPGW"
provides a workaround for dealing with broken webservice MIME responses missng the closing MIME boundary.
HermannSW 2700006U54 Tags:  datapower webcast coproc2 non-xml xslt encoding binary stylesheet ebcdic makeswa 4,358 Views
Today will be the 2nd webcast on this topic, see the dial in details under the following link
Of course you may lookup the presentation and sample files right now.
And you can listen to the audio recording next week when it will be available.
The difference if dialing in today is the Q&A session at the end of the webcast where you can state questions you have ...
HermannSW 2700006U54 Tags:  attachment xslt stylesheet dp:url-open archive zip datapower manifest 4,677 Views
provides the single-stylesheet implementation of "unzip -l" command -- nice!
HermannSW 2700006U54 Tags:  xslt hierarchy classes webcast processing non-xml data graph xml-mgmt.xsd depth-first-search heuristic 4,773 Views
Over the weekend I finished the presentation and sample files for next week Thursday webcast
"Non-XML Data Processing (2/2) Advanced Webcast on Oct 13, 2011"
Tomorrow will be the first of the webcasts, see the dial in details under the following link
"Non-XML Data Processing (1/2) Basics and Encoding Webcast on Oct 6, 2011"
This is the Agenda for tomorrow
You may find a different interesting topic in this posting and its attachment:
The question was "where to find the classes available for SOMA".
The 1st answer was "look into store:///xml-mgmt.xsd".
A better answer based on XPath expression even answers how many classes there are (varies by appliance model and firmware):
stylesheet classes.xsl which generated that diagram is attached and discussed a little bit.
You might be interested in "graph extraction" from XML data and Depth-First-Search traversal of the extracted graph
which was used to generate the "block" hierarchy diagram. Also the simple and nice heuristic which allows
to allocate exactly the vertical space needed for a node in the diagram is interesting -- in "XML speak" instead of "graph speak"
it would be "count(descendant-or-self::*)" ...
In this same title developerWorks DataPower Forum thread
the EXSLT support of several XSLT 1.0 processors is compared.
In the attached archive stylesheet exslt.xsl is used to display tabular, interlinked output detailing the support.
With DataPower firmware 3.8.2 a bunch of new EXSLT functionality has become available.
Eg. the Math package is completely implemented since then.
If you just want to lookup DataPower EXSLT support for firmware 3.8.2 and higher take this link:
These three functions ARE supported in DataPower firmware (even in pre 3.8.2 firmware) and are just incorrectly reported as unsupported:
func:function, func:result, dyn:evaluate
If you are interested in XSLT in a browser (as I) then you might want to look into this posting of the same thread:
There Firefox (the only browser with significant EXSLT support) uses the same exslt.xsl stylesheet to produce the table.
The following post even shows how serialization of XML output for display in the browser can be achieved without having access to <dp:serialize> ...
The last posting showed how to process quoted-printable encoded root section of a multipart/signed message by a simple transformation with service chaining.
* need to decode a quote-printable string
* do not like service chaining
* do not like to have a seperate WTX binary transform action with QUOTEDTOTEXT() function
then the single stylesheet solution posted here may be what you want to try out:
Attached to that posting is a correctly formatted multipart/signed sample message (with CRLFs) and the stylesheet.
This sytelsheet is a nice demonstration of these features:
* accessing Non-XML input
* UTF-8 validate input (needed for safety, OK for scenario since ASCII is a subset of UTF-8)
* having a func:function doing some preparation work (eliminate soft line breaks, split at '=3D')
* having a recursive func:function doing the rest.
Important for the recursive function is doing only what needs to be done in the stylesheet and rely on efficient extension functions (regexp:replace) where possible.
Last, but not least, it demonstartes how you can map a unicode code point to a parsed charecter entity inside a stylesheet by mapping (for ISO-8859-1 sample).