Processing JSON in XSLT can be done "as string" or via conversion to JSONX since years.
XQuery (with JSONiq extensions) can natively process XML as well as JSON.
GatewayScript can natively process JSON, and with ≥7.2.0.0 firmware natively XML as well.
Customer was on 6.0.1.5 and therefore GatewayScript was no option (≥7.0.0.0).
Solution I provided used XSLT to do backend calls by <dp:url-open> and store results into context variables, and then XQuery script to nicely combine the responses in XQuery (see above forum posting for details):
Both samples did output JSON, the small sample presented now was answering a colleagues question in how to generate XML from JSON.
This is the input and generated output:
From my initial XML-->JSONX-->JSON transformation rdp87.2.xsl:
I created "all-in-one" stylesheet for doing the XML-->JSON transformation rdp87.3.xsl:
Now I created two XML FW on a box with 4.0.2.13 firmware containing the (performance) fix for APAR IC90781 from May 2013 fixpack.
Both services used an XML Manager with "Profile Rule" in XML Manager's Compile Options policy.
Finally I did:
clear stylesheet cache for XML manager "profile"
send 1000 requests to service by
$ time for((f=1;f<=1000;++f)); do curl --data-binary @rdp87.1000.xml http://dp7-l3:port ; echo; done
switch to "Status->XML Processing->Stylesheet Profiles" in WebGUI and take screenshot
Here is Stylesheet Profiles for XML->JSONX->JSON:
And here is Stylesheet Profiles for XML->JSON:
Yes, the total time taken for rdp87.2.xsl (1961ms) in the templates is by a factor of 2.66 greater than that of rdp87.3.xsl (735ms).
But that does only make an effect if the whole operation of a service is just XML->JSON conversion.
Typically in real live services other operations like AAA, SLM actions or other will happen.
If this is the case, then the (small) runtime overhead added is outweighted by maintainability and ease of stylesheet in my eyes.
<UPDATE> rdp87.2a.xsl runtime is reduced by 15% compared to rdp87.2.xsl in making use of <xsl:import> and <xsl:apply-templates> instead of dp:transform().
Today I want to describe a method for easily creating ALL sample requests against XML management interface
by using SoapUI tool (there are free and Pro editions, the free edition is sufficient here).
First SoapUI does need access to "store:///xml-mgmt.wsdl" and the other files. In order to not having to download the files I just created a HTTP service on a 5.0.0.4 XI52 box * listening on port 88
* with Base Directory "store:///"
Now accessing "http://firestar.boeblingen.de.ibm.com:88/xml-mgmt.wsdl" is possible.
Next create a new SoapUI project
* provide a name * reference "xml-mgmt.wsdl" as Initial WSDL * keep option "Create Requests" selected
* press "OK" that's it.
This creates new project with "Request 1" under "operation":
This is the Request screen showing some choices:
The management interface provides huge number of choices -- you have to select a single one:
By right-clicking into the request you can save the generated file.
Last, but not least, the generated sample file from our Böblingen lab XI52 box:
$ echo "<foobar/>" | coproc2 Shakespeare.xsl - http://dp2-l3:2223 When Shakespeare asked, "To be, or not to be?", he did not provide the answer. But programming can. Well the answer is FF. (2B |~ 2B = FF, bitwise hexadecimal) $
</UPDATE2>
<UPDATE>
Below listings do not look that nice as they should:
despite "monospace" face option below listings do show up proportional in all browsers (not in preview modes, only when posted :-( )
Internet Explorer does display white color characters as black (see "xsl:stylesheet" after DOCTYPE in first line, "black on black" can only be made visible by selecting the region)
Without any change the output of TOhtml tool can be used in this Blog better, see next posting "binary-append.xsl" as a first example.
It is not possible to upload arbitrary files "as Image" to this Blog.
But uploading a HTML file as Image works. Now just change in HTML view
<img src="..." .../>
to
<iframe width="700" height="400" src="..."/>
that's all !
I checked that display is monospaced and no "black on black" occurs for Firefox, Chrome and Internet Explorer browsers.
</UPDATE>
I was able to use the external Datapower forum {code}...code...{code} feature in this Blog and posted about that back in last April,
"Re: How to get the developerWorks Forum {code}...{code} section in my Blog?":
But later last year some security feature change blocked the "same look" for all browsers but IE ... :-(
Now I searched for a "nice" syntax highlighting solution I want to use for code listings on this Blog. An important feature besides being "nice" was that just doing "copy&paste" should work.
I remembered the "TOhtml" feature of "vi" editor (yes, I am still using "vi" editor, in addition to gedit). Last time I used that feature was end of 2009 ...
One of the APARs fixed is IC70937 (VIEWING A REMOTE WSDL OF A WEBSERVICE PROXY IN WEBGUI FAILS IN CASE OF NON-DEFAULT XML MANAGER AND BASIC AUTHENTICATION). Part of fixing this APAR was enhancing CLI copy as well as XML management FetchFile commands in adding xmlmgr as optional new argument.
In pre-4.0.1.0 firmware releases copy/FetchFile always took the Basic Auth settings from User Agent of a domain's default XML Manager.
With 4.0.1.0 you are now able to specify the XML Manger to use for accessing its User Agent's Basic Auth settings for the copy operation. This allows you to define several XML Mangers with different User Agents to allow copy from differently secured web locations.
See the modified CLI help here:
xi50(config)# help copy copy [-f] <source-URL> <destination-URL> [xmlmgr] Uses a specified protocol (HTTP, HTTPS, SCP, or SFTP) to copy a file to or from the device. The optional "-f" argument forces an unconditional copy. With thisswitch enabled, the CLI does not warn of possible file overwrites. If optional "xmlmgr" argument is provided then User Agent is retrieved from XML Manager "xmlmgr" instead of default XML Manager for determining Basic Auth settings. ... xi50(config)#
And this is the modified FetchFile command with optional "XMLManager" element (from store:///xml-mgmt.xsd):
XML Manager "sec" references " User Agent "sec" which provides the access credentilals in its "Basic-Auth Policy" tab. Here you can see how to copy file "stock.wsdl" from secured directory http://dp4-l3/sec onto DataPower appliance:
Customer wanted to know how to store files (retrieved by scheduled rule) locally on DataPower box.
Solution consists of: * scheduled rule * dp:url-open to retrieve file * base64 encode it (either its serialization for XML, or the binaryNode response for Non-XML) * use set-file XML management operation to store the file * access XML management interface from within a stylesheet to execute the request
At the end of the posting a shell-script for conversion of a file to its corresponding XML management set-file request is provided. This is handy for use with doSoma tool.
Here file "testfile" gets stored as "local:///myTestfile2" in "default" domain ...
Hi >
> How can i count number of requests passing through the processing policy.
>
for those policies which are tied to specific stylesheets you may look
up "Status-->XML Processing-->Stylesheet Executions" in WebGUI.
You get the exact counts of stylesheet executions for 10 sec, 1 min, 10 min, 1 hour and 1 day.
You may reset the counters to 0 by flushing the corresponding XML manager's stylesheet cache.
> is there any Dp:element , XSL element or function available.
>
This seems to indicate that you want to access these counts from within a stylesheet?
You can access above mentioned counters via XML management interface:
$ doSoma admin StatusStylesheetExecutions.xml dp3-l3.boeblingen.de.ibm.com:5550 | \ > xpath++ "/*/*/*/*/*[contains(URL,'coproc2.xsl')]" - Enter host password for user 'admin': ------------------------------------------------------------------------------- <StylesheetExecutions xmlns:env="http://www.w3.org/2003/05/soap-envelope" xmlns:dp="http://www.datapower.com/schemas/management">
This posting is a complete walk-through of the steps needed to issue calls against the XML management interface from within DataPower stylesheets.
As precondition we have already enabled XML Management interface on dp3-l3:5550. Also we have created user "soma" in group "developer" with password "secret" not restricted to a domain.
We make use of dp:soap-call() extension function. Because the XML Management interface is accessed over SSL normally we have to provide an SSL-Proxy-Profile in that call. Therefore steps A-D show how to create a SSL-Proxy-Profile from scratch via WebGUI.
A) Administration-->Miscellaneous-->Crypto Tools, "Generate Key" (this is just an example)
enter CN "xslt"
click "Generate Key"
Confirm generation of key
This results in the following message:
Action completed successfully.
Generated private key on the HSM. and exported a copy in "temporary:///xslt-privkey.pem
Generated Certificate Signing Request in "temporary:///xslt.csr
Generated Self-Signed Certificate in "cert:///xslt-sscert.pem and exported a copy in "temporary:///xslt-sscert.pem
Generated a Crypto Key object named "xslt" and a Crypto Certificate object named "xslt"
B) Objects-->Crypto Configuration-->Crypto Identification Credentials
click "Add"
set "Name" to "xslt-cred"
select "xslt" as "Crypto Key"
select "xslt" as "Certificate"
click apply
Result:
C) Objects-->Crypto Configuration-->CryptoProfile
click "Add"
set "Name" to "xslt-profile"
select "xslt-cred" as "Identification Credentials"
select "xslt-profile" for "Forward (Client) Crypto Profile"
click apply
Result:
Now we are prepared for the stylesheet work.
This is the syntax of dp:soap-call(): dp:soap-call(URL, message, SSL-Proxy-Profile, flags, SOAP-action, HTTP-headers, process-faults, timeout)
We take the default for process-faults (false()) and no timeout. Since the stylesheet is executed on the box the URL is just 'https://127.0.0.1:5550/service/mgmt/current'. As message we pass in the SOMA call -- in the example here the "FirmwareVersion" SOMA call. As SSL-Proxy-Profile we just pass "xslt-ssl", that is what we created by above steps. A "0" is passed for flags as described in documentation. We do not need a SOAP-action, passing the empty string suppresses the addition of this header. Last we have to pass the HTTP headers
application/soap+xml as Content-Type
the basicAuth string with user name and password for Authorization
So find here the complete stylesheet (it is attached in this developerWorks posting):
The problem statement was: Can some one help with sample soma file to get list of MQFSH object
names and its op state and Administrative State properties alone.
In the thread it turned out that the ObjectStatus of all objects could be selected only. And it was proposed that an XPath expression needs to be applied to the SOMA response from DataPower to restrict the output to what is needed.
See here how doSoma and xpath++ tools play well together:
$ doSoma admin ObjectStatus.xml dp3-l3:5550 | xpath++ \ > "//ObjectStatus[Class='MQSourceProtocolHandler']\ > /*[contains('|OpState|AdminState|Name|',concat('|',name(),'|'))]/text()" - Enter host password for user 'admin':
------------------------------------------------------------------------------- down ------------------------------------------------------------------------------- disabled ------------------------------------------------------------------------------- MQ_FSH_1 ------------------------------------------------------------------------------- down ------------------------------------------------------------------------------- enabled ------------------------------------------------------------------------------- MQ_FSW_2 $
could have been titled "Processing binary data in DataPower stylesheets 1/x". This posting is the second of several postings on processing of binary data in DataPower stylesheets.
Again this is on CDATA as in the postings "CDATA .../x". This time the (customer) problem was as follows:
accept XML input data
do XML threat protection
if no threat, pass input unmodified to backend (preserve any CDATA elements).
While preserving of CDATA might be done with cdata-section-elements of <xsl:output ... /> you have to know which elements you want to have CDATA output for, In case of DataPowwer as generic proxy any element might contain CDATA section or might not contain CDATA sections.
If the service would have XML request type CDATA sections are gone -- therefore we need a Non-XML service.
But XML threat protection does not work in Non-XML service -- therefore we need a second XML service.
Now lets discuss the solution I came up with.
Setup XML-FW with XML request type with its own XML Manager and any specific XML Parser settings. (I did set "XML Element Depth" to 1, service is on port 2060)
Setup XML-FW with Non-XML request type, and these two actions: 1) binary transform action from INPUT to NULL with stylesheet check.xsl 2) results action from INPUT to OUTPUT
This is the binary transform stylesheet check.xsl I used.
The <dp:input-mapping ... /> uses Flat File Descriptor (FFD) store:///pkcs7-convert-input.ffd shipped with DataPower and used by five PKCS7 stylesheets located in store:/// folder. As documented in the FFD itself (see below) it generates structure <object><message>***binary data***</message></object> with a binaryNode as child of <message> element. Find the details on binaryNode in previous posting.
Now the "binary data" (which in this case is XML data, but we want pass it unmodified) needs to be posted to second XML (threat protection) service by <dp:url-open>. Posting binary data with <dp:url-open> is done with data-type="base64". Therefore we just convert the binaryNode data to base64 by dp:binary-encode(/object/message) in <dp:url-open>. We are not interested in the response of the service called but only in its response code, therefore response="responsecode". The result of <dp:url-open> call is stored in variable $reponse. In case the HTTP response code is not equal to 200 we know that something is wrong, and for the service called we know that XML threat protection found a problem. Therefore we abort service execution by <dp:reject>.
So if service execution does not get aborted by <dp:reject> the results action from INPUT to OUTPUT just copies the unmodified INPUT to OUTPUT (preserving any CDATA section). Otherwise (in case of a detected XML threat) the service returns a generic error message to the client.
What we have seen is
how to process binary (Non-XML) input data (resulting in a binaryNode)
how to convert a binaryNode to a base64 encoded string
how to pass "binary" data to dp:url-open() by data-type="base64"
how to process response code of dp:url-open
By the arguments above there cannot be a solution without a side call.
And this is from stylesheet "store:///pkcs7-convert-input.ffd":
... <!-- This FFD converts the input into an XML tree like this:
Several years ago I had the need to create many little chess problem solution animations. With some shell scripts, netpbm tools and gifsicle I was able to create animated gifs like that on the left (of size 80x80 pixels for fitting on my Siemens S55 cell phone display at that time). The chess problems were so called shortest construction tasks, have a look at "Construction task" on this page http://en.wikipedia.org/wiki/Chess_problem and you will find my animation as well as the definition and a reference to Shortest construction tasks maphttp://stamm-wilbrandt.de/chess/en/sct_map.html (the solution shown in the animation is for problem "Black mates White by changing a pawn into a knight").
I am an everyday runner since many years and searched the web for an animation, but did not find one. Therefore I created the animation on the left. You may find a detailed description on the tools used and the "howto" here: http://www.stamm-wilbrandt.de/streakrunner.de/streak-en.html
As seen above animated gifs are good for uninterrupted animations. But later I wanted some kind of "solution player" like a CD player.
Animation (play) shoud be interruptible and single stepping (back and forth) should be possible. The solution (on the left) was to have an iframe element with the following elements inside :
anim.gif (animated .gif file)
anim.html (referencing anim.gif)
uD.gif (for D=0,...,10, single position .gif file)
anim.D.html (for D=0,...,10, referencing uD.gif)
Each .html file defines an image map with the active areas making the left side controls (the well known symbols for "stop", "back to start", "single step back", ...,"play") active by defining the needed URLs to anim.html, anim.0.html, ..., anim.10.html. You may inspect the details by clicking on link anim.3.html and inspect the HTML page source.
XML
To start generating chess animations with XML first you have to create a single position picture. This can be done by a stylesheet putting pictures for each field on the chessboard together. Having a single .gif for each chess piece (bishop, king, (k)night, pawn, queen, "empty"), piece color (black/white) and field color (black/white), with naming allowing easy referencing by XSLT is helpful. Find below the .gifs "b.gif", "bbb.gif", "bbw.gif", "bkb.gif", ..., "wrb.gif", "wrw.gif":
This XML file is an encoding of the mate position of above chess animations:
85
I am an everyday runner since many years and searched the web for an animation, but did not find one. Therefore I created the animation on the left. You may find a detailed description on the tools used and the "howto" here: 
