Topic
  • 28 replies
  • Latest Post - ‏2011-06-10T12:41:28Z by SystemAdmin
SystemAdmin
SystemAdmin
6772 Posts

Pinned topic url-open for TCP server response attribute

‏2011-05-04T15:43:22Z |
The tcp server returns a string, something like "62054267|1|5|107|1304519468|test.txt".

So I set response="binary" and use the String.ffd to get the string.

The datapower gives error: Invalid response type 'binary' and the datapower process finishes there even before opening the connection to TCP server. see case (2) below.

When changing the response attribute to ignore, tcp server successfully returns a string, but the datapower process hangs. case(1) below

When removing the response attribute, tcp server successfully returns a string, but the datapower assumes the response is xml and complains about the first character of the response.

Question: What response attribute should I use?

Error in three cases (1), (2) and (3)

(1) response="ignore":

<dp:url-open target="{$tcp-url}" response="ignore" data-type="xml">

- The tcp server successfully returns the string but the datapower process hangs during the response processing.

(2) response="binary"

<dp:url-open target="{$tcp-url}" response="binary" data-type="xml">
mpgw (testMPG-FS): url-open: Invalid response type 'binary' when fetching url 'tcp://10.119.74.90:80/'

- it fails before opening connection

(3) no response attribute

<dp:url-open target="{$tcp-url}" data-type="xml">
mpgw (testMPG-FS): illegal character '6' at offset 0 of tcp://10.119.74.90:80/

- The tcp server successfully returns the string : "62054267|1|5|107|1304519468|test.txt"
But the datapower seems to expect xml and complains about the first character of the response, 6.

-----my xsl-----

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:dp="http://www.datapower.com/extensions" extension-element-prefixes="dp" exclude-result-prefixes="dp"
>

<dp:output-mapping href="String.ffd" type="ffd"/>

<xsl:template match="/">
<xsl:variable name="tcp-url" select="'tcp://10.119.74.90:80'" />
<xsl:variable name="commit">
<commit><![CDATA0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]>
</commit>
</xsl:variable>

<xsl:variable name="response">
<dp:url-open target="{$tcp-url}" response="binary" data-type="xml">
<xsl:value-of select="$commit" />
</dp:url-open>
</xsl:variable>
<Conversion>
<str>
<xsl:value-of select="$response"/>
</str>
</Conversion>

<xsl:variable name="text" select="/Conversion/str/text()" />
<dp:set-variable name="'var://context/ECM/text'" value="$text" />

<dp:set-variable name="'var://service/routing-url'" value="'http://services-sitce1.bankofamerica.com:80/wsi/FNCEWS35SOAP/'" />

</xsl:template>
</xsl:stylesheet>

Updated on 2011-06-10T12:41:28Z at 2011-06-10T12:41:28Z by SystemAdmin
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-05T17:44:21Z  
    Hello,

    first you should have placed your code inside a {​code} section like this
    
    
    {​code
    } your code goes unmodified here 
    {​code
    }
    
    to avoid scrambling the code (entering the {​code} tag is tricky, as in my posting from yesterday zero white space helped again).

    OK, I made your stylesheet completely working.

    First step is set up a reliable TCP server.
    I always use netcat tool for that (as in similar thread https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14598154&#14598154).

    First start the TCP server with the intended response setup:
    echo "62054267|1|5|107|1304519468|test.txt" | nc -l 8080

    This is the complete output of one-shot TCP server:
    
    [stammw@dp4-l3 ~]$ echo 
    "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 request [stammw@dp4-l3 ~]$
    


    And this is the netcat client just for testing that TCP server works (no network issues).
    Your intended response is returned:
    
    $ echo 
    "request" | nc dp4-l3.boeblingen.de.ibm.com 8080 62054267|1|5|107|1304519468|test.txt $
    

    Now some changes to your stylesheet were needed.

    The first was only needed because I wanted to use coproc2 tool and that cannot send
    stylesheet and FFD file together -- just fetch the FFD from a webserver is the solution.
    
    $ diff orig.xsl tcp.xsl 7c7 < <dp:output-mapping href=
    "String.ffd" type=
    "ffd"/> --- > <dp:output-mapping href=
    "http://dp4-l3.boeblingen.de.ibm.com/String.ffd" type=
    "ffd"/>
    


    Now I needed to access my "netcat TCP server":
    
    10c10 < <xsl:variable name=
    "tcp-url" select=
    "'tcp://10.119.74.90:80'" /> --- > <xsl:variable name=
    "tcp-url" select=
    "'tcp://dp4-l3.boeblingen.de.ibm.com:8080'" />
    


    Then I needed to fix the CDATA problem in your posting without {​code} section:
    
    12c12 < <commit><![CDATA0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]> --- > <commit><![CDATA[0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]]>
    


    Finally, there is no response type "binary", it is "binaryNode", see dp:url-open:
    
    17c17 < <dp:url-open target=
    "{$tcp-url}" response=
    "binary" data-type=
    "xml"> --- > <dp:url-open target=
    "{$tcp-url}" response=
    "binaryNode" data-type=
    "xml"> $
    


    Running this stylesheet against the netcat TCP server works ...
    
    [stammw@dp4-l3 ~]$ echo 
    "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0 [stammw@dp4-l3 ~]$
    


    ... but produces "***BINARY NODE***" output:
    
    $ java coproc2 tcp.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223 ***BINARY NODE***-1text/xml $
    


    Next step is how to deal with binaryNode data inside a stylesheet.

    This is a posting on DataPower proprietary node type "binaryNode" (**):
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14572518&#14572518

    These are relevant postings/slides for accessing binaryNode data by dp:url-open
    https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/tags/dp:url-open()
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119&aid=1#page=8
    So this little change makes your stylesheet fully functional:
    
    $ diff tcp.xsl tcp2.xsl 23c23 < <xsl:value-of select=
    "$response"/> --- > <xsl:value-of select=
    "dp:decode(dp:binary-encode($response/result/binary),'base-64')"/> $
    


     
    I just learned recently myself that using "dp:decode(_,'base-64')" has an important
    advantage over using "dp:-binary-decode(_)" in (**) posting:
    "dp:decode(_,'base-64')" does UTF-8 validation!!
    See for the clumsy solution I provided in (**) posting for UTF-8 validation (regexp).

     
    So now this is your modified stylesheet in action:
    
    [stammw@dp4-l3 ~]$ echo 
    "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0 [stammw@dp4-l3 ~]$
    


    
    $ java coproc2 tcp2.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223 62054267|1|5|107|1304519468|test.txt $
    


     
    For completeness the FFD file, the stylesheet is attached:
    
    $ cat String.ffd <?xml version=
    "1.0" encoding=
    "UTF-8" ?> <File version=
    "2.1" name=
    "Conversion"> <Field name=
    "str" type=
    "String" /> </File> $
    


     
    Hermann<myXsltBlog/>

    Attachments

  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-05T18:06:59Z  
    • HermannSW
    • ‏2011-05-05T17:44:21Z
    Hello,

    first you should have placed your code inside a {​code} section like this
    <pre class="jive-pre"> {​code } your code goes unmodified here {​code } </pre> to avoid scrambling the code (entering the {​code} tag is tricky, as in my posting from yesterday zero white space helped again).

    OK, I made your stylesheet completely working.

    First step is set up a reliable TCP server.
    I always use netcat tool for that (as in similar thread https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14598154&#14598154).

    First start the TCP server with the intended response setup:
    echo "62054267|1|5|107|1304519468|test.txt" | nc -l 8080

    This is the complete output of one-shot TCP server:
    <pre class="jive-pre"> [stammw@dp4-l3 ~]$ echo "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 request [stammw@dp4-l3 ~]$ </pre>

    And this is the netcat client just for testing that TCP server works (no network issues).
    Your intended response is returned:
    <pre class="jive-pre"> $ echo "request" | nc dp4-l3.boeblingen.de.ibm.com 8080 62054267|1|5|107|1304519468|test.txt $ </pre>
    Now some changes to your stylesheet were needed.

    The first was only needed because I wanted to use coproc2 tool and that cannot send
    stylesheet and FFD file together -- just fetch the FFD from a webserver is the solution.
    <pre class="jive-pre"> $ diff orig.xsl tcp.xsl 7c7 < <dp:output-mapping href= "String.ffd" type= "ffd"/> --- > <dp:output-mapping href= "http://dp4-l3.boeblingen.de.ibm.com/String.ffd" type= "ffd"/> </pre>

    Now I needed to access my "netcat TCP server":
    <pre class="jive-pre"> 10c10 < <xsl:variable name= "tcp-url" select= "'tcp://10.119.74.90:80'" /> --- > <xsl:variable name= "tcp-url" select= "'tcp://dp4-l3.boeblingen.de.ibm.com:8080'" /> </pre>

    Then I needed to fix the CDATA problem in your posting without {​code} section:
    <pre class="jive-pre"> 12c12 < <commit><![CDATA0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]> --- > <commit><![CDATA[0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]]> </pre>

    Finally, there is no response type "binary", it is "binaryNode", see dp:url-open:
    <pre class="jive-pre"> 17c17 < <dp:url-open target= "{$tcp-url}" response= "binary" data-type= "xml"> --- > <dp:url-open target= "{$tcp-url}" response= "binaryNode" data-type= "xml"> $ </pre>

    Running this stylesheet against the netcat TCP server works ...
    <pre class="jive-pre"> [stammw@dp4-l3 ~]$ echo "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0 [stammw@dp4-l3 ~]$ </pre>

    ... but produces "***BINARY NODE***" output:
    <pre class="jive-pre"> $ java coproc2 tcp.xsl some.xml http: //dp3-l3.boeblingen.de.ibm.com:2223 ***BINARY NODE***-1text/xml $ </pre>

    Next step is how to deal with binaryNode data inside a stylesheet.

    This is a posting on DataPower proprietary node type "binaryNode" (**):
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14572518&#14572518

    These are relevant postings/slides for accessing binaryNode data by dp:url-open
    https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/tags/dp:url-open()
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119&aid=1#page=8
    So this little change makes your stylesheet fully functional:
    <pre class="jive-pre"> $ diff tcp.xsl tcp2.xsl 23c23 < <xsl:value-of select= "$response"/> --- > <xsl:value-of select= "dp:decode(dp:binary-encode($response/result/binary),'base-64')"/> $ </pre>

     
    I just learned recently myself that using "dp:decode(_,'base-64')" has an important
    advantage over using "dp:-binary-decode(_)" in (**) posting:
    "dp:decode(_,'base-64')" does UTF-8 validation!!
    See for the clumsy solution I provided in (**) posting for UTF-8 validation (regexp).

     
    So now this is your modified stylesheet in action:
    <pre class="jive-pre"> [stammw@dp4-l3 ~]$ echo "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0 [stammw@dp4-l3 ~]$ </pre>

    <pre class="jive-pre"> $ java coproc2 tcp2.xsl some.xml http: //dp3-l3.boeblingen.de.ibm.com:2223 62054267|1|5|107|1304519468|test.txt $ </pre>

     
    For completeness the FFD file, the stylesheet is attached:
    <pre class="jive-pre"> $ cat String.ffd <?xml version= "1.0" encoding= "UTF-8" ?> <File version= "2.1" name= "Conversion"> <Field name= "str" type= "String" /> </File> $ </pre>

     
    Hermann<myXsltBlog/>
    I just looked into getting the stylesheet working.

    Now that I look again and I do not understand why you are using a dp:output-mapping.
    It does work without, too:
    $ java coproc2 tcp3.xsl some.xml http://dp3-l3.boeblingen.de.ibm.com:2223
    62054267|1|5|107|1304519468|test.txt
    $ 
    $ diff tcp2.xsl tcp3.xsl 
    7,8d6
    < <dp:output-mapping href="http://dp4-l3.boeblingen.de.ibm.com/String.ffd" type="ffd"/>
    < 
    21,22d18
    < <Conversion>
    < <str>
    24,25d19
    < </str>
    < </Conversion>
    $
    


     
    Hermann <myXsltBlog/>
    Updated on 2014-03-25T03:31:32Z at 2014-03-25T03:31:32Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-05T19:03:03Z  
    Thank you for attaching the modified xsl. I tested using the modified code. I received a string "62054276|1|5|120|1304619813|test.txt" from the TCP server and I do not see the error in system log, but the process hangs.
    Do you know what causes the process to hang? The response gets to the datapower but something prevents the datapower from processing the response (sommehow cannot parse the response and the process stops).
    My datapower firmware is 3.8.0.12 and I got the response option: "binary" from Extensions-Common.pdf below
    (Now this is incorrect)
    Syntax:
    <dp:url-open
    target=”url”
    response=”xml | binary | ignore | responsecode | responsecode-ignore | savefile”

    Please advise what next step I should take.
    ----
    Below are odd things I found during the course of testing.
    I set data-type="xml"

    (1) This works (2 lines)
    <commit><![CDATAxxxx]>
    </commit>

    (2) With this, the datapower generates two requests to TCP server (3 lines)
    <commit>
    <![CDATAxxxx]>
    </commit>

    (3) This will generate error before opening the connection to TCP server, the datapower does not recognize it as xml (one line)
    <commit><![CDATAxxxx]></commit>

    (4) strip all tags off the well-formed xml and only send values
    <doc>
    <FileName>test</FileName>
    <FileExt>txt</FileExt>
    </doc>

  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-05T22:31:15Z  
    And one more odd thing is this:

    We changed the response of the TCP server to return xml (<docID>xxx</docID> ) instead of the bar separated string for just a testing purpose. So in the xsl, the response attribute was changed to xml (response="xml").
    The TCP server successfully returned <docID>xxx</docID>, but the datapower hung during the response processing, just like the binary response.

    I was wondering if you could take a look at the source code of the url-open for the protocol of tcp.

    When I tested the protocol of ftp for other occasions, the process did not hang.

  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-06T19:02:43Z  
    datapower process hanging problem:

    It turned out that the datapower is not closing the tcp connection. The client (datapower) opens the connection, so is the client supposed to close the connection? Anway, we have to finish this proof of concept, so for now we made a change so that the tcp server closes the connection. After this change I do not see the process hanging.
    The TCP server returns a string, but somehow the response string is not properly handled.
    I use below code and the context variable ends up an empty string.

    <xsl:variable name="response">
       <dp:url-open target="{$tcp-url}" response="binaryNode" data-type="xml">
          <xsl:value-of select="$commit" />
       </dp:url-open>
    </xsl:variable>
    <Conversion>
    <str>
    <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/>
    </str>
    </Conversion>
     
    <xsl:variable name="text" select="/Conversion/str/text()" />
    <dp:set-variable name="'var://context/ECM/text'" value="string($text)" />
    
    Updated on 2014-03-25T03:31:22Z at 2014-03-25T03:31:22Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-06T21:31:54Z  
    I found that select="dp:decode(dp:binary-encode($response/result/binary),'base-64')" works but accessing Conversion/str does not.
    So I just use the decode(binary-encode) value.

    <!-- binary-encode and decode works but accessing Conversion/str does  not work -->
    <Conversion>
    <str>
    <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/>
    </str>
    </Conversion>
     
    <!-- below line does not work and text will end up an empty string -->
    <xsl:variable name="text" select="/Conversion/str/text()" />
    <dp:set-variable name="'var://context/ECM/text'" value="string($text)" />
     
    <!-- use below -->
    <xsl:variable name="responseString">
    <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/>
    </xsl:variable>
     
    <dp:set-variable name="'var://context/ECM/responseString'" value="string($responseString)" />
    
    Updated on 2014-03-25T03:31:17Z at 2014-03-25T03:31:17Z by iron-man
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-07T00:47:08Z  
    datapower process hanging problem:

    It turned out that the datapower is not closing the tcp connection. The client (datapower) opens the connection, so is the client supposed to close the connection? Anway, we have to finish this proof of concept, so for now we made a change so that the tcp server closes the connection. After this change I do not see the process hanging.
    The TCP server returns a string, but somehow the response string is not properly handled.
    I use below code and the context variable ends up an empty string.

    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr"><xsl:variable name="response"> <dp:url-open target="{$tcp-url}" response="binaryNode" data-type="xml"> <xsl:value-of select="$commit" /> </dp:url-open> </xsl:variable> <Conversion> <str> <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/> </str> </Conversion> <xsl:variable name="text" select="/Conversion/str/text()" /> <dp:set-variable name="'var://context/ECM/text'" value="string($text)" /> </pre>
    > It turned out that the datapower is not closing the tcp connection. The client (datapower) opens the connection, so is the client
    > supposed to close the connection?
    You are right, the server has to close the TCP connection.
    For XML DataPower would know when the end of message is reached.
    For Non-XML the end of message indicator is the closing of the TCP connection.
    If the server does not close the connection DataPower does not know whether additional data might get sent or not.

    So dp:url-open waits until connection gets closed or timeout.
    You may set timeout attribute of dp:url-open.
    (see posting Adding latency during service development for backend simulation for an interesting application of the timeout value)

    > Anway, we have to finish this proof of concept, so for now we made a change so that the tcp server
    > closes the connection. After this change I do not see the process hanging.
    > ...
    Good.

     
    Hermann<myXsltBlog/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-09T19:46:07Z  
    Please confirm that timeout attribute to url-open works.
    I tested the timeout attribute by generating a datapower hanging condition
    (use <xsl:value-of select="'xxx'" instead of <![CDATA xxx ]>)
    The timeout is set to 5 sec.
    With the modified commit variable, TCP server does not close the connection.
    In 5 sec after the connection is established, the datapower is supposed to close the connection.
    But it does not.
    Would you please test the timeout attribute in your environment, and let me know the result

    <xsl:variable name="commit">
    <commit><xsl:value-of select="'0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0'" />
    </commit>
    </xsl:variable>
     
    <xsl:variable name="response">
    <dp:url-open target="{$tcp-url}" response="binaryNode" timeout="5" data-type="xml">
    <xsl:value-of select="$commit" />
    </dp:url-open>
    </xsl:variable>
    
    Updated on 2014-03-25T03:31:07Z at 2014-03-25T03:31:07Z by iron-man
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-10T01:15:02Z  
    Please confirm that timeout attribute to url-open works.
    I tested the timeout attribute by generating a datapower hanging condition
    (use <xsl:value-of select="'xxx'" instead of <![CDATA xxx ]>)
    The timeout is set to 5 sec.
    With the modified commit variable, TCP server does not close the connection.
    In 5 sec after the connection is established, the datapower is supposed to close the connection.
    But it does not.
    Would you please test the timeout attribute in your environment, and let me know the result

    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr"><xsl:variable name="commit"> <commit><xsl:value-of select="'0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0'" /> </commit> </xsl:variable> <xsl:variable name="response"> <dp:url-open target="{$tcp-url}" response="binaryNode" timeout="5" data-type="xml"> <xsl:value-of select="$commit" /> </dp:url-open> </xsl:variable> </pre>
    Hello,

    there is no big difference between your previous version with CDATA and <xsl:value-of ...>.

    But you are right, the timeout attribute of <dp:url-open ...> does not work for tcp://... connections.

    I tested that with just starting netcat listening on port 8080, without any echo command before.
    In this case netcat displays the (commit) string received from DataPower and DataPower waits for input entered in the shell where "nc -l 8080" was started.
    But DataPower waits and waits and does not timeout after 5 seconds.

    Please see this posting on a working timeout scenario:
    https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/entry/adding_latency_during_service_development_for_backend_simulation94

    The difference was using the HTTP protocol instead of tcp://.

    Now I changed your stylesheet to use the HTTP protocol against a sleep CGI script on a webserver:
    
    $ cat /var/www/cgi-bin/sleep.pl #!/usr/bin/perl   print 
    "echo Content-Type: text/plain\n\n"; print 
    "Hello ..."; sleep(15); print 
    "... world!\n"; $
    


    Here is the diff of your stylesheet to the previous version:
    
    $ diff tcp3.xsl tcp4.xsl 8c8 < <xsl:variable name=
    "tcp-url" select=
    "'tcp://dp4-l3.boeblingen.de.ibm.com:8080'" /> --- > <xsl:variable name=
    "tcp-url" select=
    "'http://dp4-l3.boeblingen.de.ibm.com/cgi-bin/sleep.pl'" /> 10c10 < <commit><![CDATA[0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]]> --- > <commit><xsl:value-of select=
    "'0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0'" /> 15c15 < <dp:url-open target=
    "{$tcp-url}" response=
    "binaryNode" data-type=
    "xml"> --- > <dp:url-open target=
    "{$tcp-url}" response=
    "binaryNode" timeout=
    "5" data-type=
    "xml"> $
    


    Running this against the sleep service times out after 5 seconds:
    
    $ time java coproc2 tcp4.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223   real    0m5.401s user    0m0.360s sys     0m0.122s $
    


    And changing the sleep value from 15 to 1 works as expected:
    
    $ time java coproc2 tcp4.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223 Hello ...... world!   real    0m1.374s user    0m0.331s sys     0m0.115s $
    


    Please create a PMR on timeout attribute of <dp:url-open ...> not working for tcp://.

    Either
    • this will be fixed (likely)
    • or at least the documentation will point out that timeout does not work for tcp:// (and why)

     
    Hermann<myXsltBlog/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-10T12:00:41Z  
    Thank you for testing and confirming that the timeout attribute works with url-open for http, but not with the one for tcp.
    I'm using 3.8.0.12, which is the latest under the 3.8.0.x branch released in late April 2011, so I think this is a bug.
    I've also noticed this. After datapower hangs and I kill the process by cntl-C, datapower stops writing transactions in the probe.
    This occurs not only for this transaction but also all subsequent transactions.
    And it impacts all domains on the same device.
    To recover from this, domain restart does not work and device restart is needed.
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-10T12:53:35Z  
    Question about closing connection:

    What will happen if network problem arises right after tcp communication is established from datapower to the TCP server, and datapower does not have the timeout set.

    (1) Datapower will hang forever becasue TCP server cannot send a close connection signal back to the datapower?

    (2) Underlining os will send network error back to the datapower, and the datapower will not hang?

    (3) Datapower will be aware of the network problem somehow and it will not hang?

    Please let me know.
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-10T18:59:11Z  
    I opened a PMR: 59398 499 000-timeout attribute for tcp url-open does not work.
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-12T18:34:08Z  
    I opened a PMR: 59398 499 000-timeout attribute for tcp url-open does not work.
    > I opened a PMR: 59398 499 000-timeout attribute for tcp url-open does not work.
    >
    Thank you for doing that -- it is really needed.

    Regarding your questions, I started two requests and waited and waited ... and finally stopped them after half an hour.
    Both showed up as established in "Status-->IP-Network-->TCP Port Status" the whole time.

     
    Now let me take the opportunity to riddle you/the other readers of this DataPower Forum:
    Based on the information in this thread there exists a (simple) workaround
    on DataPower side to use <dp:url-open ...> in a stylesheet and access a
    rawTCP backend and get a timeout before PMR 59398,499,000 gets fixed.

    This shows the workaround in action with a timeout of 8 seconds:
    $ time java coproc2 tcp5.xsl some.xml http://dp3-l3.boeblingen.de.ibm.com:2223
     
    real    0m8.313s
    user    0m0.374s
    sys     0m0.103s
    $
    


    Have fun ;-)

     
    Hermann <myXsltBlog/>
    Updated on 2014-03-25T03:30:34Z at 2014-03-25T03:30:34Z by iron-man
  • Liv2luv
    Liv2luv
    573 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-12T19:36:17Z  
    • HermannSW
    • ‏2011-05-12T18:34:08Z
    > I opened a PMR: 59398 499 000-timeout attribute for tcp url-open does not work.
    >
    Thank you for doing that -- it is really needed.

    Regarding your questions, I started two requests and waited and waited ... and finally stopped them after half an hour.
    Both showed up as established in "Status-->IP-Network-->TCP Port Status" the whole time.

     
    Now let me take the opportunity to riddle you/the other readers of this DataPower Forum:
    Based on the information in this thread there exists a (simple) workaround
    on DataPower side to use <dp:url-open ...> in a stylesheet and access a
    rawTCP backend and get a timeout before PMR 59398,499,000 gets fixed.

    This shows the workaround in action with a timeout of 8 seconds:
    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr">$ time java coproc2 tcp5.xsl some.xml http://dp3-l3.boeblingen.de.ibm.com:2223 real 0m8.313s user 0m0.374s sys 0m0.103s $ </pre>

    Have fun ;-)

     
    Hermann <myXsltBlog/>
    Just a guess, will any one of these two time out parameters make the trick ?

    var://local/_extension/timeout
    var://service/mpgw/backend-timeout

    Thanks.
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-12T20:26:37Z  
    Please post your tcp5.xsl.

    The TCP call using url-open is a side call and I use loopbackFirewall as the main backend
    (The loopback firewall is set to 'var://service/routing-url')

    The previous responder is mentioning the service's backned value:

    Q1: Is he correct?
    Q2: If correct, does it work for the side call or just for the main backend call?

  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-12T21:12:09Z  
    • Liv2luv
    • ‏2011-05-12T19:36:17Z
    Just a guess, will any one of these two time out parameters make the trick ?

    var://local/_extension/timeout
    var://service/mpgw/backend-timeout

    Thanks.
    Hi Suresh,

    > Just a guess, will any one of these two time out parameters make the trick ?
    >
    > var://local/_extension/timeout
    > var://service/mpgw/backend-timeout
    >
    No, the PMR was created because
    <dp:url-open target="'tcp://...'" ...>...</dp:url-open>
    

    hangs indefinitely when TCP server does not close the connection, regardless what timeout values get specified.

     
    Hermann <myXsltBlog/>
    Updated on 2014-03-25T03:30:29Z at 2014-03-25T03:30:29Z by iron-man
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-12T21:45:16Z  
    Please post your tcp5.xsl.

    The TCP call using url-open is a side call and I use loopbackFirewall as the main backend
    (The loopback firewall is set to 'var://service/routing-url')

    The previous responder is mentioning the service's backned value:

    Q1: Is he correct?
    Q2: If correct, does it work for the side call or just for the main backend call?

    > Please post your tcp5.xsl.
    >
    I will post, but later after others had a chance to provide a solution.

    I responded the solution back to you in the PMR and by email because this is urgent to you.

    > The TCP call using url-open is a side call and I use loopbackFirewall as the main backend
    > (The loopback firewall is set to 'var://service/routing-url')
    >
    > The previous responder is mentioning the service's backned value:
    >
    > Q1: Is he correct?
    No.

    > Q2: If correct, does it work for the side call or just for the main backend call?
    >
    Hermann<myXsltBlog/>
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-16T22:44:50Z  
    OK, no new ideas for the workaround.

    Let me give a hint:
    Making use of
    <dp:url-open target="http://..." response="binaryNode" timeout="5">
    

    with working timeout (see above) is the basis of the workaround.
    How can the access to remote TCP server (tcp://...) be done based on this?

     
    Hermann <myXsltBlog/>
    Updated on 2014-03-25T03:30:06Z at 2014-03-25T03:30:06Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-17T13:52:05Z  
    • HermannSW
    • ‏2011-05-16T22:44:50Z
    OK, no new ideas for the workaround.

    Let me give a hint:
    Making use of
    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr"><dp:url-open target="http://..." response="binaryNode" timeout="5"> </pre>
    with working timeout (see above) is the basis of the workaround.
    How can the access to remote TCP server (tcp://...) be done based on this?

     
    Hermann <myXsltBlog/>
    Hermann

    I will be testing this very problem in a couple of weeks. So this added 'timeout' attribute actually does end the tcp connection? If so, great news. This particular issue has been causing the connections to build up, and hang the boxes eventually. I automated some things for the client at silent hours, which solved the issue, like firmware reloads (crazy, yes, but fairly quick and painless). My thinking was to utilize a stateful raw tcp handler. Currently they are using an ssl proxy to make these transactions, which is not preferable. It's basically TLS over TCP, and the connections build, stay established, and then the box hangs/and/or reboots as I understand it. But if this timeout actually ends the connection, then I will definitely go that route first, vs. the stateful raw handler.

    Thanks for the feedback -
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-23T13:47:10Z  
    Hermann

    I will be testing this very problem in a couple of weeks. So this added 'timeout' attribute actually does end the tcp connection? If so, great news. This particular issue has been causing the connections to build up, and hang the boxes eventually. I automated some things for the client at silent hours, which solved the issue, like firmware reloads (crazy, yes, but fairly quick and painless). My thinking was to utilize a stateful raw tcp handler. Currently they are using an ssl proxy to make these transactions, which is not preferable. It's basically TLS over TCP, and the connections build, stay established, and then the box hangs/and/or reboots as I understand it. But if this timeout actually ends the connection, then I will definitely go that route first, vs. the stateful raw handler.

    Thanks for the feedback -
    Kirby,
    >
    > I will be testing this very problem in a couple of weeks. So this added 'timeout' attribute actually does end the tcp connection?
    > If so, great news. This particular issue has been causing the connections to build up, and hang the boxes eventually. I automated
    > some things for the client at silent hours, which solved the issue, like firmware reloads (crazy, yes, but fairly quick and
    > painless). My thinking was to utilize a stateful raw tcp handler. Currently they are using an ssl proxy to make these transactions,
    > which is not preferable. It's basically TLS over TCP, and the connections build, stay established, and then the box hangs/and/or
    > reboots as I understand it. But if this timeout actually ends the connection, then I will definitely go that route first, vs. the
    > stateful raw handler.
    >

    the timeout attribute should work for "tcp:///..." connections (for dp:url-open() as well as as static backend of a MPGW).
    But it does not until (new) PMR 59398,499,000 (IC76418) has been fixed.

    I do not completely understand your SSL proxy problem.
    Perhaps best would be if you create a new thread stating the problem first and then your ideas on how to workaround the problem with tcp:///.

     
    Hermann<myXsltBlog/>
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-23T14:49:36Z  
    • HermannSW
    • ‏2011-05-23T13:47:10Z
    Kirby,
    >
    > I will be testing this very problem in a couple of weeks. So this added 'timeout' attribute actually does end the tcp connection?
    > If so, great news. This particular issue has been causing the connections to build up, and hang the boxes eventually. I automated
    > some things for the client at silent hours, which solved the issue, like firmware reloads (crazy, yes, but fairly quick and
    > painless). My thinking was to utilize a stateful raw tcp handler. Currently they are using an ssl proxy to make these transactions,
    > which is not preferable. It's basically TLS over TCP, and the connections build, stay established, and then the box hangs/and/or
    > reboots as I understand it. But if this timeout actually ends the connection, then I will definitely go that route first, vs. the
    > stateful raw handler.
    >

    the timeout attribute should work for "tcp:///..." connections (for dp:url-open() as well as as static backend of a MPGW).
    But it does not until (new) PMR 59398,499,000 (IC76418) has been fixed.

    I do not completely understand your SSL proxy problem.
    Perhaps best would be if you create a new thread stating the problem first and then your ideas on how to workaround the problem with tcp:///.

     
    Hermann<myXsltBlog/>
    OK,

    no further submission after giving the hint, and I have been asked to provide the solution.

     
    As stated in the hint the workaround makes use of the fact that timeout attribute works for http protocol, see tcp4.xsl in this posting:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14620016#14614888
    For the riddle
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14620016#14616340

    this is small diff between tcp4.xsl and tcp5.xsl:
    
    $ diff tcp4.xsl tcp5.xsl 8c8 < <xsl:variable name=
    "tcp-url" select=
    "'http://dp4-l3.boeblingen.de.ibm.com/cgi-bin/sleep.pl'" /> --- > <xsl:variable name=
    "tcp-url" select=
    "'http://127.0.0.1:8081'" /> 15c15 < <dp:url-open target=
    "{$tcp-url}" response=
    "binaryNode" timeout=
    "5" data-type=
    "xml"> --- > <dp:url-open target=
    "{$tcp-url}" response=
    "binaryNode" timeout=
    "8" data-type=
    "xml"> $
    


    Just go to an internal HTTP service listening on port 8081.

    The last piece needed for the workaround now is a HTTP2rawTCP service on port 8081.

    In posting "Processing of Non-XML rawTCP data is possible on DataPower appliances"
    http://www.ibm.com/developerworks/forums/thread.jspa?threadID=364522

    a rawTCP2HTTP service was provided.
    Now this is how you can setup a HTTP2rawTCP service:
    • create a MPGW
    • HTTP FSH listening on 127.0.0.1:8081
    • pass-thru request type
    • pass-thru response type
    • "default" policy
    • static backend type
    • backend URL "tcp://dp4-l3.boeblingen.de.ibm.com:8081"

    That's it -- this service just
    • takes HTTP request
    • sends rawTCP to backend
    • receives rawTCP response from backend
    • returns HTTP response

    Because of the timeout set for the (local) http-connection the workaround is complete.
    (see Figure 7 from Multi-Protocol Gateway configuration)

     
    Hermann<myXsltBlog/>
  • Liv2luv
    Liv2luv
    573 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-23T19:57:33Z  
    • HermannSW
    • ‏2011-05-23T14:49:36Z
    OK,

    no further submission after giving the hint, and I have been asked to provide the solution.

     
    As stated in the hint the workaround makes use of the fact that timeout attribute works for http protocol, see tcp4.xsl in this posting:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14620016#14614888
    For the riddle
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14620016#14616340

    this is small diff between tcp4.xsl and tcp5.xsl:
    <pre class="jive-pre"> $ diff tcp4.xsl tcp5.xsl 8c8 < <xsl:variable name= "tcp-url" select= "'http://dp4-l3.boeblingen.de.ibm.com/cgi-bin/sleep.pl'" /> --- > <xsl:variable name= "tcp-url" select= "'http://127.0.0.1:8081'" /> 15c15 < <dp:url-open target= "{$tcp-url}" response= "binaryNode" timeout= "5" data-type= "xml"> --- > <dp:url-open target= "{$tcp-url}" response= "binaryNode" timeout= "8" data-type= "xml"> $ </pre>

    Just go to an internal HTTP service listening on port 8081.

    The last piece needed for the workaround now is a HTTP2rawTCP service on port 8081.

    In posting "Processing of Non-XML rawTCP data is possible on DataPower appliances"
    http://www.ibm.com/developerworks/forums/thread.jspa?threadID=364522

    a rawTCP2HTTP service was provided.
    Now this is how you can setup a HTTP2rawTCP service:
    • create a MPGW
    • HTTP FSH listening on 127.0.0.1:8081
    • pass-thru request type
    • pass-thru response type
    • "default" policy
    • static backend type
    • backend URL "tcp://dp4-l3.boeblingen.de.ibm.com:8081"

    That's it -- this service just
    • takes HTTP request
    • sends rawTCP to backend
    • receives rawTCP response from backend
    • returns HTTP response

    Because of the timeout set for the (local) http-connection the workaround is complete.
    (see Figure 7 from Multi-Protocol Gateway configuration)

     
    Hermann<myXsltBlog/>
    Thank you for the solution. It was interesting.

    So, to implement this workaround, we need to have service chaining:

    MPG-A (http) <-> (http) MPG-B (tcp) <-> TCP Server

    In this pattern, When the connection to TCP server is broken after the connection is established (MPG-B to TCP), datapower connection would still hang in MPG-B ? Please validate.

    Thanks.
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-23T21:38:20Z  
    • Liv2luv
    • ‏2011-05-23T19:57:33Z
    Thank you for the solution. It was interesting.

    So, to implement this workaround, we need to have service chaining:

    MPG-A (http) <-> (http) MPG-B (tcp) <-> TCP Server

    In this pattern, When the connection to TCP server is broken after the connection is established (MPG-B to TCP), datapower connection would still hang in MPG-B ? Please validate.

    Thanks.
    Hi Suresh,

    > Thank you for the solution. It was interesting.
    >
    > So, to implement this workaround, we need to have service chaining:
    >
    > MPG-A (http) <-> (http) MPG-B (tcp) <-> TCP Server
    >
    > In this pattern, When the connection to TCP server is broken after the connection is established (MPG-B to TCP), datapower connection would still hang in MPG-B ? Please validate.
    >
    thank you for this very good question.

    I just tried it and this time I prefixed the netcat command with time command:
    
    [stammw@dp4-l3 ~]$ time nc -l 8081 0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0 real    2m18.165s user    0m0.000s sys     0m0.000s [stammw@dp4-l3 ~]$
    


    I just sent a single request against this "broken" TCP service:
    
    $ time java coproc2 tcp5.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223   real    0m8.588s user    0m0.338s sys     0m0.058s $
    


    After netcat timeout I did this experiment again, but this time I took a packet capture on DataPower.
    Exactly after 120 seconds DataPower did send a [FIN, ACK] to the TCP server.

    So PMR 59398,499,000 (IC76418) is just about timeout being always 120 seconds instead of being configurable.

    Anyone making use of this solution now before IC76418 is fixed needs to make sure by SLM configuration
    that at most 500 requests per second are accepted for such a TCP service in order to not run out of free ports
    before the waiting connections timeout after 2 minutes.

     
    Hermann<myXsltBlog/>
  • HermannSW
    HermannSW
    6388 Posts

    Re: url-open for TCP server response attribute

    ‏2011-05-24T08:25:12Z  
    • HermannSW
    • ‏2011-05-23T21:38:20Z
    Hi Suresh,

    > Thank you for the solution. It was interesting.
    >
    > So, to implement this workaround, we need to have service chaining:
    >
    > MPG-A (http) <-> (http) MPG-B (tcp) <-> TCP Server
    >
    > In this pattern, When the connection to TCP server is broken after the connection is established (MPG-B to TCP), datapower connection would still hang in MPG-B ? Please validate.
    >
    thank you for this very good question.

    I just tried it and this time I prefixed the netcat command with time command:
    <pre class="jive-pre"> [stammw@dp4-l3 ~]$ time nc -l 8081 0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0 real 2m18.165s user 0m0.000s sys 0m0.000s [stammw@dp4-l3 ~]$ </pre>

    I just sent a single request against this "broken" TCP service:
    <pre class="jive-pre"> $ time java coproc2 tcp5.xsl some.xml http: //dp3-l3.boeblingen.de.ibm.com:2223 real 0m8.588s user 0m0.338s sys 0m0.058s $ </pre>

    After netcat timeout I did this experiment again, but this time I took a packet capture on DataPower.
    Exactly after 120 seconds DataPower did send a [FIN, ACK] to the TCP server.

    So PMR 59398,499,000 (IC76418) is just about timeout being always 120 seconds instead of being configurable.

    Anyone making use of this solution now before IC76418 is fixed needs to make sure by SLM configuration
    that at most 500 requests per second are accepted for such a TCP service in order to not run out of free ports
    before the waiting connections timeout after 2 minutes.

     
    Hermann<myXsltBlog/>
    During my morning run to the IBM lab I thought about what I wrote last night.

    > ...
    > After netcat timeout I did this experiment again, but this time I took a packet capture on DataPower.
    > Exactly after 120 seconds DataPower did send a [FIN, ACK] to the TCP server.
    >
    The reason for terminating the connection after 2 minutes was the Back Side Timeout of the workaround MPGW.

    > So PMR 59398,499,000 (IC76418) is just about timeout being always 120 seconds instead of being configurable.
    >
    This is not correct, as stated in this posting above tcp:/// connections hang indefinitely for dp:url-open.
    That is what the PMR is about.

    > Anyone making use of this solution now before IC76418 is fixed needs to make sure by SLM configuration
    > that at most 500 requests per second are accepted for such a TCP service in order to not run out of free ports
    > before the waiting connections timeout after 2 minutes.
    >
    Now this can be handled much better as I found out during above mentioned run.
    Just set Back Side Timeout of workaround MPGW to eg. 9 seconds in case of dp:url-open timeout of 8 seconds to local workaround MPGW.
    This way the dp:url-open will timeout after 8 seconds, and the connection from workaround MPGW to TCP server times out after 9 seconds!
    I tested it and it really works that way.
    This way SLM only needs to restrict access to the service at 7000 requests per second to prevent running out of free ports.

    Here is a little ASCII art representation of the whole dribble-workaround solution:
    
    .                              MPGW toRawTCPwithTimeout +---------------------------------------------------------+ !                                                         ! !    /-----------------------------------------------\    ! !    ! ...                                           !    ! !    ! <dp:url-open target=
    "'http://127.0.0.1:8081'" !    ! !    !   response=
    "binaryNode" timeout=
    "8">          !    ! !    !   ...                                         !    ! !    ! </dp:url-open>                                !    ! !    ! ...                                           !    ! !    \-----------------------------------------------/    ! !                                       !                 ! !                                       !                 ! !  +---------------------------------------------------+  ! !  !                                   NULL            !  ! !  !                                                   !  !      /  ------> !  !  Match( Url = * ) --- xform( local:/​/​/tcp5.xsl )  !  ! ----/---> !  !                                                   !  !    /  !  !                                   NULL            !  ! !  +---------------------------------------------------+  ! !                                                         ! !                                                         ! !                                                         ! <------ !                        (pass-thru)                      ! !                                                         ! +---------------------------------------------------------+         MPGW workaround +---------------+ HTTP FSH     !               !  tcp:
    //dp4-l3.boeblingen.de.ibm.com:8081 ---------------> !  (pass-thru)  ! ----------------------------------------> 127.0.0.1:8081  !               !        ( Back Side Timeout = 9 ) !               ! <--------------- !  (pass-thru)  ! <---------------------------------------- !               ! +---------------+
    

    Hermann&lt;myXsltBlog/>