In addition the posting shows how to make use of netcat tool as simplest form of a rawTCP server. This
is useful in debugging issues between DataPower and other TCP servers
-- get it working with netcat and you know that DataPower side is fine.
rawTCP2HTTP never was a perfect solution, but things have become really bad:
While the passthru/passthru trick with FSH timeout of 1sec works for clients not sending a FIN with rawTCP2HTTP running on <18.104.22.168 firmwares (there DataPower did not send a FIN to client when FSH timeout occured), networking behavior changed starting with 22.214.171.124 firmware. Now connection is terminated completely on frontside timeout, resulting in an Internal error response to client.
What remains is that sending rawTCP binary message from client terminated with FIN (like netcat command "nc" does) still works even with 7.0.0.x firmware.
There were Requests For Enhancement, but I cannot find any of them anymore in RFE tool :-(
In writing this up for a WSTE webcast below solution for rawTCP Non-XML processing became nearly trivial(!).
And now it only makes use of definitely supported features (there is nothing besides passthru and raw XML Handler).
See the details on this slide (packet capture screenshot on the following slide):
since quite some time there exists an Enhancement request for a rawTCP Non-XML Front Side Handler
1.5 years ago I developed a prototype of a rawTCP2HTTP converter in DataPower Firmware which allows to convert
rawTCP Non-XML data to HTTP chunked data which can then be processed by any (DataPower) HTTP service
(see IBM publication Bridging raw TCP binary data to HTTP, http://priorartdatabase.com/IPCOM/000197959)
So it seems that processing rawTCP binary data is not possible with DataPower appliances.
Even one week ago I would have said "it is impossible to process Non-XML rawTCP data on DataPower appliances".
But that is not true, and this posting is just another one in the series of "does not work out-of-the-box on DataPower but can be made working"
So now that we know that it can be done the question is how.
It turns out to be really easy to create a rawTCP2HTTP service as MPGW service on DataPower.
Even though XA35 and XS40 do not allow for binary data processing they at least allow for Non-XML UTF-8 data processing by the first link of above list.
We make use of the fact that binary input of Non-XML transformations needs to be "consumed", otherwise the default behavior is that it will be copied to the output (see 3rd and 1st link above).
In addition we use a rawTCP XML Front Side Handler as that is the only one not complaining about protocol violations.
And then we make sure that INPUT is guaranteed to not being consumed -- thats all.
So this is rawTCP2HTTP MPGW service:
Front Side Protocol: Stateless Raw XML Handler listening on port1
Request Type: Non-XML
Type: static backend
Backend URL: http://127.0.0.1:port2
Response Type: Pass-Thru
And this is rawTCP2HTTP policy:
Results action from NULL to OUTPUT
In the example port1 is 2091 and port2 is 2092.
Sending data with a HTTP POST to DataPower can be easily done with with cURL or ApacheBench tool:
curl --data-binary @file ... http://dpbox:port1
ab -p file ... http://dpbox:port1
Sending rawTCP data can be done by Netcat tool (nc), this statement sends file to service listening on port at host:
nc host port < file
To have a basis for measurements I chose a very simple Non-XML example service (toHexb.xsl).
It just returns the (Non-XML) input as hexadecimal string, see posting DataPower "binary" node type (binaryNode) for details:
Now sending a request is not that spectacular -- we get the same hexadecimal output for file te3t we can also get by octal dump (od) tool:
[stammw@bl3d2027@de ~]$ nc 192.168.10.1 2091 <te3t ; echo
[stammw@bl3d2027@de ~]$ od -Ax -tcx1 te3t
000000 t e 003 t
74 65 03 74
But this is really over rawTCP!
See the following screenshot of a "All Interfaces" (3.8.2 firmware) packet capture.
The Filter for ports 2091 and 2092 just selects the "interesting" part of packect capture.
At the bottom left you see the command execution and output.
On the bottom right you see Follow TCP Stream for services on port 2091 (rawTCP) and 2092 (HTTP).
Hexadecimally displayed content of packet 116 is just the HTTP post of 2091 service to 2092 service after FIN ACK has been received.
This is the first restriction of rawTCP2HTTP service -- it cannot deal with HTTP persistent connections.
Time stamps for packets 110 and 128 show that the transaction took 2.003ms.
But that is not the value which is important.
Important is the average transaction overhead of rawTCP service on 2091 over HTTP service on 2092.
There are quite some tools like ApacheBench (ab) mentioned above for getting performance information of HTTP services.
I was not able to find a similar tool for benchmarking rawTCP performance.
Therefore I wrote one myself based on TCPClient.c (http://en.wikipedia.org/wiki/Berkeley_sockets#Client).
It was able to benchamrk rawTCP as well as HTTP data (by sending the necessary HTTP Post data).
For persistent and non-persistent connection HTTP benchmarking my numbers were similar to ApacheBench results (-k option for "keepalive").
These are the average results for sending 20000 requests with a concurrency of 20 to a 9004 DataPower from a directly connected Laptop:
0.1985ms per request for rawTCP service on 2091
0.0809ms per request for HTTP service on 2092 (without persistent connections)
0.0647ms per request for HTTP service on 2092 (with persistent connections)
So the overhead of rawTCP2HTTP service is 0.1985ms - 0.0647ms = 0.1338ms.
While for this very simple example service the overhead time dominates the total transaction time, 0.14ms overhead for rawTCP processing enablement is a good price for many real world Non-XML processing scenarios -- until now this is the only way to process rawTCP Non-XML data on DataPower appliances.
These are reasons why the "real" Non-XML rawTCP Front Side Handler of above mentioned Enhancement Request is still needed:
enabling persistent connections
reducing/eliminating 0.1338ms overhead of adding/removing of HTTP headers
having to use "XML rawTCP Front Side Handler" for dealing with "Non-XML data" sounds "interesting" at least :-)
Last, but not least, see that NonXML rawTCP processing on a XA35 or XS40 DataPower appliance is possible by above technique and
the technique from 1st link posting above (by "prepend=" trick with convert-http action):
$ od -Ax -tcx1 test
000000 t e s t 1 2 3 t e s t 1 2 3 t e
74 65 73 74 31 32 33 74 65 73 74 31 32 33 74 65
000010 s t \n
73 74 0a
$ nc cosmopolitan 2091 < test | tidy -q -xml
<?xml version="1.0" encoding="utf-8"?>
$ soma/doSoma admin soma/version.xml cosmopolitan:5550 | xpath++ "//Version/text()" -
Enter host password for user 'admin':
I tested rawTCP2HTTP on all supported release branches.
Finally I just tested it on a box with 126.96.36.199 firmware without problems.
188.8.131.52 DataPower Firmware was released in 1H2009 -- since then rawTCP2HTTP service waited to be discovered ... ;-)