If you do access your DataPower boxes via a VNC session as
described over your company VPN from remote, your traffic from VPN
server to your VNC server is not encrypted. These days end-to-end
encryption should be done everywhere [remote-->VPN
server-->VNC server--(https)-->DataPower]. One option for
this is to tunnel the VNC connection over SSH. There are many
tutorials (it is really simple), this
is one of them.
I wanted to be able to do same or similar [of course store as file and view via gimp or browser is possible].
Back in 2011 I gave 2 WSTE webcasts on "Non-XML Data Processing in WebSphere DataPower SOA Appliances Stylesheets". The 2nd webcast shows on slide 28 how I did convert a bitmap image into textual output making use of the Braille Patterns.
This is the conversion for snowman.pbm, .pbm is portable bitmap format from netpbm tools:
Typically only the top 2x3 dots of 2x4 get used, as you can see above I used all 2x4.
samples.txt[.pre.html] contains various sample output produced (shown below), which are part of pbmtobraille.c's comments too.
9x9.pbm is a really crazy parsing sample according the .pbm spec, following this statement:
"Mr. Poskanzer cautions that programs that read this format should be as
lenient as possible, accepting anything that looks remotely like a pixmap."
This is the header section demonstrating basic use with pbmtext output, including negation of generated output, as well as the help line telling the tool's features:
This is top of tail comment section, showing graphviz output done by pbmtobraille:
And finally this is bottom section showing a bigger layout in vertical direction (layout=TB is default, Top to Bottom):
OK, lets first see what the Text clock from Ross Goodwin looks like after being ported to GatewayScript on DataPower. More details can be found in posting Brilliant: Text Clock displays current time using text from public domain books. As you can see in the 50% resolution screenshot below the page states the time in big bold red characters, "the time is", "eight", "fourty-two", "in the morning". Additionally the DataPower port reports the maximal time taken by asynchronous url.open() calls.
There were several problems to be resolved in porting, described one by one now, all changes can be looked up in this side-by-side diff. MPGW service export rgTextClock.zip allows you to easily play with (access it at "http://yourBox:3600" in your browser)..
The script needs access to "urlopen" module, header variables (in order to set "text/html" Content-Type for the generated response) and service variable (in order to set 'var://service/mpgw/skip-backside' making rgTextClock MPGW service a loopback service). In addition it needs to simulate write access to HTML elements identified by ID. This is done by object "o" acting as associative array. Varaible "t" and constant "d" of 50 millisecond are used in function "check()" discussed further below:
Now we have to solve the problem on how long to wait until all asynchronous "urlopen.open()" calls have finished, the same functionality as provided by DataPower's event-sink action on rule level. This is done by function "check()" that gets called with a 50ms delay. If "all is done", it just calls function "result()" to generate and return the HTML response page. Otherwise it just calls itself again, after another delay of 50ms. When is "all is done" achieved? In the case we just have to count the number of completed HTML elements writte, or the number of keys of object "o". If it is 7, we are done. Here you can see function "check()" together with its initial invocation:
For completeness this is function "result()" that
generates response HTML page (includes a 15sec page refresh for clock updates in your browser)
sets "text/html" Content-Type header
does skip-backside for a loopback service
If you watch the "completed in" times reported below the main response you will notice quite some variance. Btw, since the data is accessed from Ross' web server your DataPower box needs internet access in order to work.
The refresh every 15 seconds will show 4 different pages per minute, but the time displayed will be the same for a given minute.
I was really impressed and thought how to make it run on DataPower. There is no "sleep" or "delay" function among the DataPower extension functions for XSLT/XQuery/JSONiq besides the <dp:url-open> timeout trick (for development purposes only!). So I looked at GatewayScript and remembered my previous posting on GatewayScript timers, and this posting shows how to do sleepsort based on GatewayScript timers, but some tricks are needed.
The first problem that needed to be solved was how to pass an argument to a (anonymous) timeout function -- that was solved easily via some googleing, just pass arg1 [, arg2, ...] after the timeout value (in milliseconds) in setTimeout().
Next problem specifically to setTimeout solution for sleepsort was how to output the result [by session.output.write()] after the last timeout has fired. My first solution was to setup an additional timeout longer than all values to be sorted, but without knowing the maximum value that was difficult, and not nice.
The next idea was better, making use of the fact that only the last session.output.write() wins in sending the response back, just define two timeouts for each number to be sorted:
timeout at x millisecond for number x, and timeout function appending x to result array when timeout fires
timeout at x+1 millisecond writing the (current) result to session.output
OK, now we have 2N timeouts for sorting N numbers, but the final result gets written last and therefore returned.
Of course there is the problem that sorting might not be correct if too many numbers get sorted and race condifions between timeouts might occur. Interestingly sorting 20000 numbers works always fine in GatewayScript(!).
I did pipeing the output of sleepsort.js into sleepsort.check.js in order to find occurences of incorrect sortings:
Originally conceived by mathematician David Hilbert in the 1920s, this famous thought experiment challenges thinkers to truly understand what it means to reach infinity. Imagine working at a hotel with an infinite number of rooms available for rent.
I posted this on twitter:
#math ∞ room hotel fully booked, ∞ busses each with a countably ∞ number of passengers arrive. Added comment to video http://t.co/JrRDhRoAMS
This posting is on how to connect a serial console server with DataPower XG45/XI52/XB62/IDG appliances.
Why do we want to do this? It allows remote access to serial console of DataPower appliance and seeing all bootup messages. For example I can login to "myXI52" appliance over serial line from anywhere in the world by:
$ telnet dp-gport 10005
Connected to dp-gport.
Escape character is '^]'.
Welcome to DataPower XI52 console configuration.
Copyright IBM Corporation 1999-2014
Version: XI188.8.131.52.0 build main.252668 on Oct 22, 2014 4:51:19 PM
Serial number: 6805302
Notice: startup config contains errors.
Whether you are allowed to access DataPower serial line at all and/or remotely depends on your company's security policy.
We have serial console servers in both Böblingen lab DataPower racks (one for Level 2 support and one for my Level 3 support rack). Since three weeks now we have a "DataPower room" in Böblingen lab -- August, Andreas and I share an office (see room sign at bottom of this posting). We do share appliances as well. This is the 16port serial console server in my rack:
On the left "ETHERNET WAN PORT" provides the network connectivity, and lines 1-4 provide serial (RS232) access to 4 of my DataPower appliances.
The serial line connection of "myXI52" appliance until today was done by connecting the blue serial line cable received with my appliance in April last year with the grey serial-to-RJ45 cable I used to connect a XI50 appliance to serial console server before.
The black cable does the same now, and is built after this prototype from Andreas, see his dwAnswer posting for details. In short you just have to cross the green and green-white cables, that's all:
The last time I did such "hardware stuff" myself was when tuning my Asuro robot back in 2006 ;-)
I wanted to play with it a bit, and wanted to deal with parallel timeouts. "Timeout demo" GatewayScript td.js is the result. Depending on the input timeout values in milliseconds the 1st timecout kicks in first and cancels 2nd timer, or 2nd timeout kicks in and overwrites GS response (only the very last that is written to session.output gets returned as response):
Not surprisingly the stylesheet did not compile, it missed "version", "encoding" and other stuff required by XSLT 1.0 spec and did not know what "𝖛𝖊𝖗𝖘𝖎𝖔𝖓" and "𝖊𝖓𝖈𝖔𝖉𝖎𝖓𝖌" should be. So I repeated compile-fix until the stylesheet finally compiled again. This is the result -- vi's syntax highlighting does not like black-letter ;-)
And wow, the output was BIG and very detailed. I found many functions not mentioned in above util function list, and wanted to get a compressed list. Therefore I did just set showHidden to false, here you can see the output:
From a DataPower support perspective I have to say that any use of undocumented functionality is not supported.
But util.inspect() is a useful tool at least for debugging, your self developed modules as well as required 3rd party modules.
In CLI debugger the length of output string is restricted to 512 bytes by default, but can be overridden by -L option (-1 meaning unbounded). So here you see that you can do util.inspect() in CLI debugger as well:
xi52(config)# show debug-actions
Session ID Transaction ID Service Name File Location Remote Address In Use Remote User User Location Elapsed Time ---------- -------------- ------------ ------------- -------------- ------ ----------- ------------- ------------ 391 298305 coproc2gatewayscript local:///temp.js 184.108.40.206 No 00:00:05
but I did find no others. Here let me define that I mean "higher Message count" by "more active" (58692/53367/31819).
On the weekend I finally got the right idea on how to write an IBM forum crawler, so let me first copy in the result here before discussing the crawler below. As can be seen there are far more than 4000 IBM forums, and DataPower forum is 5th most active. The generated table only displays entries with at least 5-digit message count number:
This is Firefox only because any one of the big5 browsers working is enough and I deveoped it with Firefox (and its web console).
There is exactly one warning left
"Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help http://xhr.spec.whatwg.org/ IBM-Forum-crawler.js.html:30"
which is no problem here because the main browser page gets updated for each of the (369) category pages with current information like this:
Using Firefox web console on a IBM category page I was able to find the access to all information needed in the page. The "data picking" auxiliary functions are generated from that investigations,
Function strSort(a, b) is responsible for descending sorting of (found) forum lines by Message count.
The actual crawling is done by a simple recursive Depth First Search (DFS) starting on root IBM forum category page. These IBM category pages do not contain cycles and therefore there was no risk on running into endless loops. Anyway associative array visited implements cycle detection and abort in case you want to apply the crawler somewhere else.
Avoiding CORS (Cross-origin resource sharing) problems is most easily done by running the IBM-Forum-crawler.js.html page from the same domain (www.ibm.com) as the requested resources (the forum category pages visited during DFS). IBMers like me caanot just "store" something on www.ibm.com -- but the link above shows the easy workaround I found; generate a dummy blog posting here on developerWorks, click "Insert Image" igon, select a "*.html" (!) file and click OK. I do use this trick for ages now, for all my syntax highlighted source code listings like below being an <iframe> with @src pointing to locally uploaded HTML page. This is the link to just the syntax highlighted source code page reference in below <iframe>: https://www.ibm.com/developerworks/community/blogs/HermannSW/resource/BLOGS_UPLOADED_IMAGES/IBM-Forum-crawler.js.html.html
Another strange issue is that I could not get arr.concat(A) working and therefore worked around it with another for-loop.
Currently on vacation I tried to post on DataPower forum. This was impossible on my (backlevel) 2.3.7 Android with all browsers I tried (built in, Opera mini, Dolphin, ...). This status did not change with 4.1.2 Android of my son. Logging in to write this blog posting was impossible. And posting on oeis.org was difficult because normal edit just does not work. Only editing internal format worked.
Two days I borrowed the IPOD from my son and used myAndroid as wireless access point for it only. The IPOD (standard) Safari browser just worked to my surprise, allowing to answer some DataPower forum posting, writing this blog post and submitting nice new stuff to oeis.org on 9/10.
For me this is the first time I have to use Safari browser - and using it feels good (nice animation of multiple open windows).
[the links work only if opening SVG under above link, forum software disables the links]
I did test that SVG with FF/Chrome/Opera under Linux and with big5 browsers under Win7.
Screenshot below just for browsers not capable of displaying SVG.
I have no idea why now Internet Explorer does not display the posting (it did before).
So even with Internet Explorer 10 SVG still seems to be unreliable, please open above link.
The whole thread discussed that handcrafted FFDs are not supported and referred back to this 2012 posting on the options. It also listed the only Enhancement Request that has been done since 2007 for FFD processing (FFD PMRs were fixed of course).
Further below I will show how easily binary data processing can be done with GatewayScript (available with 220.127.116.11 firmware). But before lets summarize all DataPower Non-XML data processing options here in one place:
Contivo FFDs: you need Contivo Analyst product,
no handcrafted FFDs are supported (you cannot raise PMRs against handmade FFDs)
"Binary data processing without DataGlue license!" technique, with
... <hexbin>...<hexbin> --(XSLT)-- base64 ...
... <hexbin>...<hexbin> --(XQuery)-- base64 ... (with 18.104.22.168 or higher firmware, allows for XPath 2.0)
One comment on option 4. While this works without Dataglue license (these days a XG45 without DIM option) you have to "pay" the price in form of added latency and memory consumption of the attachment processing needed by that technique.
The simple GatewayScript data structure for processing binary data is the buffer object.
For reading binary input we use readAsBuffer() method, and its documentation tries to move people to use the Buffers object.
When contexts are small, use the readAsBuffer() function. Use the readAsBuffers() function when a context is large. The readAsBuffer() function requires a contiguous memory allocation. The readAsBuffer() function is faster but is more demanding on the memory system. The readAsBuffers() function does not require contiguous memory to populate the Buffers object.
Use of Buffers might be valid for some Non-XML processing, but when the application needs access to whole input I prefer buffer.
Good news is that the first (workable) Non-XML sample program can be found in readAsBuffer() documentation itself. It is a binary identity operation with error handling. Here you can see rAB.js:
Since binary identity is not that interesting lets see now the binary reverse operation from "... without Dataglue license" posting. Adding 5 lines to rAB.js does the job. here is reverse.js:
Now lets see what both do on sample input from "... without DataGlue license" posting:
$ od -tcx1 0in0put0 0000000 \0 2 b | ~ 2 b - \0 t h a t i 00 32 62 7c 7e 32 62 20 2d 00 74 68 61 74 20 69 0000020 s t h e q u e s t i o n \0 73 20 74 68 65 20 71 75 65 73 74 69 6f 6e 00 0000037 $ $ coproc2 rAB.js 0in0put0 http://dp2-l3:2227 -s | od -tcx1 0000000 \0 2 b | ~ 2 b - \0 t h a t i 00 32 62 7c 7e 32 62 20 2d 00 74 68 61 74 20 69 0000020 s t h e q u e s t i o n \0 73 20 74 68 65 20 71 75 65 73 74 69 6f 6e 00 0000037 $ $ coproc2 reverse.js 0in0put0 http://dp2-l3:2227 -s | od -tcx1 0000000 \0 n o i t s e u q e h t s i 00 6e 6f 69 74 73 65 75 71 20 65 68 74 20 73 69 0000020 t a h t \0 - b 2 ~ | b 2 \0 20 74 61 68 74 00 2d 20 62 32 7e 7c 62 32 00 0000037 $
OK, that was small input, lets process 10MB.
$ head --bytes 10M /dev/urandom > out $ du -sb out 10485760 out $ $ time ( coproc2 reverse.js out http://dp2-l3:2227 -s > out.2 )
real 0m1.048s user 0m0.012s sys 0m0.080s $ time ( coproc2 reverse.js out.2 http://dp2-l3:2227 -s > out.3 )
real 0m1.064s user 0m0.008s sys 0m0.075s $ time ( coproc2 rAB.js out.2 http://dp2-l3:2227 -s > out.4 )
real 0m0.294s user 0m0.014s sys 0m0.074s $ $ diff out out.3 $ diff out.2 out.4 $ $ sha1sum out out.2 8fe128844bc9a19aac275272e243a1c4ce6adc7b out 66590beec5c0e226ba9efc8436d119589a67e9d8 out.2 $
Last question to be answered is on the runtime of rAB.js and reverse.js on the 10MB input. That can be answered easily based on the ExtLatency logging target of coproc2gatewayscript again blog posting:
So the reverse operation on 10MB data (read binary data, revert, output result to context) took (908-137)=771msec.
The binary identity operation on 10MB data (read binary data, output input to context) took (147-134)=13msec.
number = [ minus ] int [ frac ] [ exp ]
decimal-point = %x2E ; .
digit1-9 = %x31-39 ; 1-9
e = %x65 / %x45 ; e E
exp = e [ minus / plus ] 1*DIGIT
frac = decimal-point 1*DIGIT
int = zero / ( digit1-9 *DIGIT )
minus = %x2D ; -
plus = %x2B ; +
zero = %x30 ; 0
number ::= ( minus )? int ( frac )? ( exp )?
decimal-point ::= #x2E
DIGIT ::= #x30 - #x39
digit1-9 ::= #x31 - #x39
e ::= #x65 | #x45
exp ::= e ( minus | plus )? DIGIT+
frac ::= decimal-point DIGIT+
int ::= ( zero | digit1-9 DIGIT* )
minus ::= #x2D
plus ::= #x2B
zero ::= #x30
<?xml version='1.0'?> <disclaimer> <p>The opinions represented herein represent those of the individual and should not be interpreted as official policy endorsed by this organization.</p> </disclaimer>
Now it is clear what should happen if opening document.xml.
$ echo "<foobar/>" | coproc2 xinclude.demo.xsl - http://dp2-l3:2223 ; echo <?xml version="1.0" encoding="UTF-8"?> <document xmlns:xi="http://www.w3.org/2001/XInclude"> <p>120 Mz is adequate for an average home user.</p> <disclaimer> <p>The opinions represented herein represent those of the individual and should not be interpreted as official policy endorsed by this organization.</p> </disclaimer> </document> $
Just use "func:document(_)" instead of "document(_)", that is all that is needed:
And this is xinclude.xsl implementing "Basic Inclusion" only from XInclude spec:
Some words of caution:
xinclude.xsl does not copy comments, you will have to modify if you want to keep them
xinclude.xsl does not detect infinite xi:include loops
absolute URIs work as expected
relative URIs get interpreted relative to the storage location of "xinclude.xsl" (!)
[that is where the document() function gets executed inside func:document()].
So you may want to copy it to your local filesystem.
"/.." is not allowed for document() function in local: filesystem