Topic
28 replies Latest Post - ‏2011-06-10T12:41:28Z by SystemAdmin
SystemAdmin
SystemAdmin
6772 Posts
ACCEPTED ANSWER

Pinned topic url-open for TCP server response attribute

‏2011-05-04T15:43:22Z |
The tcp server returns a string, something like "62054267|1|5|107|1304519468|test.txt".

So I set response="binary" and use the String.ffd to get the string.

The datapower gives error: Invalid response type 'binary' and the datapower process finishes there even before opening the connection to TCP server. see case (2) below.

When changing the response attribute to ignore, tcp server successfully returns a string, but the datapower process hangs. case(1) below

When removing the response attribute, tcp server successfully returns a string, but the datapower assumes the response is xml and complains about the first character of the response.

Question: What response attribute should I use?

Error in three cases (1), (2) and (3)

(1) response="ignore":

<dp:url-open target="{$tcp-url}" response="ignore" data-type="xml">

- The tcp server successfully returns the string but the datapower process hangs during the response processing.

(2) response="binary"

<dp:url-open target="{$tcp-url}" response="binary" data-type="xml">
mpgw (testMPG-FS): url-open: Invalid response type 'binary' when fetching url 'tcp://10.119.74.90:80/'

- it fails before opening connection

(3) no response attribute

<dp:url-open target="{$tcp-url}" data-type="xml">
mpgw (testMPG-FS): illegal character '6' at offset 0 of tcp://10.119.74.90:80/

- The tcp server successfully returns the string : "62054267|1|5|107|1304519468|test.txt"
But the datapower seems to expect xml and complains about the first character of the response, 6.

-----my xsl-----

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:dp="http://www.datapower.com/extensions" extension-element-prefixes="dp" exclude-result-prefixes="dp"
>

<dp:output-mapping href="String.ffd" type="ffd"/>

<xsl:template match="/">
<xsl:variable name="tcp-url" select="'tcp://10.119.74.90:80'" />
<xsl:variable name="commit">
<commit><![CDATA0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]>
</commit>
</xsl:variable>

<xsl:variable name="response">
<dp:url-open target="{$tcp-url}" response="binary" data-type="xml">
<xsl:value-of select="$commit" />
</dp:url-open>
</xsl:variable>
<Conversion>
<str>
<xsl:value-of select="$response"/>
</str>
</Conversion>

<xsl:variable name="text" select="/Conversion/str/text()" />
<dp:set-variable name="'var://context/ECM/text'" value="$text" />

<dp:set-variable name="'var://service/routing-url'" value="'http://services-sitce1.bankofamerica.com:80/wsi/FNCEWS35SOAP/'" />

</xsl:template>
</xsl:stylesheet>

Updated on 2011-06-10T12:41:28Z at 2011-06-10T12:41:28Z by SystemAdmin
  • HermannSW
    HermannSW
    4369 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-05T17:44:21Z  in response to SystemAdmin
    Hello,

    first you should have placed your code inside a {​code} section like this
    
    
    {​code
    } your code goes unmodified here 
    {​code
    }
    
    to avoid scrambling the code (entering the {​code} tag is tricky, as in my posting from yesterday zero white space helped again).

    OK, I made your stylesheet completely working.

    First step is set up a reliable TCP server.
    I always use netcat tool for that (as in similar thread https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14598154&#14598154).

    First start the TCP server with the intended response setup:
    echo "62054267|1|5|107|1304519468|test.txt" | nc -l 8080

    This is the complete output of one-shot TCP server:
    
    [stammw@dp4-l3 ~]$ echo 
    "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 request [stammw@dp4-l3 ~]$
    


    And this is the netcat client just for testing that TCP server works (no network issues).
    Your intended response is returned:
    
    $ echo 
    "request" | nc dp4-l3.boeblingen.de.ibm.com 8080 62054267|1|5|107|1304519468|test.txt $
    

    Now some changes to your stylesheet were needed.

    The first was only needed because I wanted to use coproc2 tool and that cannot send
    stylesheet and FFD file together -- just fetch the FFD from a webserver is the solution.
    
    $ diff orig.xsl tcp.xsl 7c7 < <dp:output-mapping href=
    "String.ffd" type=
    "ffd"/> --- > <dp:output-mapping href=
    "http://dp4-l3.boeblingen.de.ibm.com/String.ffd" type=
    "ffd"/>
    


    Now I needed to access my "netcat TCP server":
    
    10c10 < <xsl:variable name=
    "tcp-url" select=
    "'tcp://10.119.74.90:80'" /> --- > <xsl:variable name=
    "tcp-url" select=
    "'tcp://dp4-l3.boeblingen.de.ibm.com:8080'" />
    


    Then I needed to fix the CDATA problem in your posting without {​code} section:
    
    12c12 < <commit><![CDATA0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]> --- > <commit><![CDATA[0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]]>
    


    Finally, there is no response type "binary", it is "binaryNode", see dp:url-open:
    
    17c17 < <dp:url-open target=
    "{$tcp-url}" response=
    "binary" data-type=
    "xml"> --- > <dp:url-open target=
    "{$tcp-url}" response=
    "binaryNode" data-type=
    "xml"> $
    


    Running this stylesheet against the netcat TCP server works ...
    
    [stammw@dp4-l3 ~]$ echo 
    "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0 [stammw@dp4-l3 ~]$
    


    ... but produces "***BINARY NODE***" output:
    
    $ java coproc2 tcp.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223 ***BINARY NODE***-1text/xml $
    


    Next step is how to deal with binaryNode data inside a stylesheet.

    This is a posting on DataPower proprietary node type "binaryNode" (**):
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14572518&#14572518

    These are relevant postings/slides for accessing binaryNode data by dp:url-open
    https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/tags/dp:url-open()
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119&aid=1#page=8
    So this little change makes your stylesheet fully functional:
    
    $ diff tcp.xsl tcp2.xsl 23c23 < <xsl:value-of select=
    "$response"/> --- > <xsl:value-of select=
    "dp:decode(dp:binary-encode($response/result/binary),'base-64')"/> $
    


     
    I just learned recently myself that using "dp:decode(_,'base-64')" has an important
    advantage over using "dp:-binary-decode(_)" in (**) posting:
    "dp:decode(_,'base-64')" does UTF-8 validation!!
    See for the clumsy solution I provided in (**) posting for UTF-8 validation (regexp).

     
    So now this is your modified stylesheet in action:
    
    [stammw@dp4-l3 ~]$ echo 
    "62054267|1|5|107|1304519468|test.txt" | nc -l 8080 0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0 [stammw@dp4-l3 ~]$
    


    
    $ java coproc2 tcp2.xsl some.xml http:
    //dp3-l3.boeblingen.de.ibm.com:2223 62054267|1|5|107|1304519468|test.txt $
    


     
    For completeness the FFD file, the stylesheet is attached:
    
    $ cat String.ffd <?xml version=
    "1.0" encoding=
    "UTF-8" ?> <File version=
    "2.1" name=
    "Conversion"> <Field name=
    "str" type=
    "String" /> </File> $
    


     
    Hermann<myXsltBlog/>

    Attachments

    • HermannSW
      HermannSW
      4369 Posts
      ACCEPTED ANSWER

      Re: url-open for TCP server response attribute

      ‏2011-05-05T18:06:59Z  in response to HermannSW
      I just looked into getting the stylesheet working.

      Now that I look again and I do not understand why you are using a dp:output-mapping.
      It does work without, too:
      $ java coproc2 tcp3.xsl some.xml http://dp3-l3.boeblingen.de.ibm.com:2223
      62054267|1|5|107|1304519468|test.txt
      $ 
      $ diff tcp2.xsl tcp3.xsl 
      7,8d6
      < <dp:output-mapping href="http://dp4-l3.boeblingen.de.ibm.com/String.ffd" type="ffd"/>
      < 
      21,22d18
      < <Conversion>
      < <str>
      24,25d19
      < </str>
      < </Conversion>
      $
      


       
      Hermann <myXsltBlog/>
      Updated on 2014-03-25T03:31:32Z at 2014-03-25T03:31:32Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-05T19:03:03Z  in response to SystemAdmin
    Thank you for attaching the modified xsl. I tested using the modified code. I received a string "62054276|1|5|120|1304619813|test.txt" from the TCP server and I do not see the error in system log, but the process hangs.
    Do you know what causes the process to hang? The response gets to the datapower but something prevents the datapower from processing the response (sommehow cannot parse the response and the process stops).
    My datapower firmware is 3.8.0.12 and I got the response option: "binary" from Extensions-Common.pdf below
    (Now this is incorrect)
    Syntax:
    <dp:url-open
    target=”url”
    response=”xml | binary | ignore | responsecode | responsecode-ignore | savefile”

    Please advise what next step I should take.
    ----
    Below are odd things I found during the course of testing.
    I set data-type="xml"

    (1) This works (2 lines)
    <commit><![CDATAxxxx]>
    </commit>

    (2) With this, the datapower generates two requests to TCP server (3 lines)
    <commit>
    <![CDATAxxxx]>
    </commit>

    (3) This will generate error before opening the connection to TCP server, the datapower does not recognize it as xml (one line)
    <commit><![CDATAxxxx]></commit>

    (4) strip all tags off the well-formed xml and only send values
    <doc>
    <FileName>test</FileName>
    <FileExt>txt</FileExt>
    </doc>

  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-05T22:31:15Z  in response to SystemAdmin
    And one more odd thing is this:

    We changed the response of the TCP server to return xml (<docID>xxx</docID> ) instead of the bar separated string for just a testing purpose. So in the xsl, the response attribute was changed to xml (response="xml").
    The TCP server successfully returned <docID>xxx</docID>, but the datapower hung during the response processing, just like the binary response.

    I was wondering if you could take a look at the source code of the url-open for the protocol of tcp.

    When I tested the protocol of ftp for other occasions, the process did not hang.

  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-06T19:02:43Z  in response to SystemAdmin
    datapower process hanging problem:

    It turned out that the datapower is not closing the tcp connection. The client (datapower) opens the connection, so is the client supposed to close the connection? Anway, we have to finish this proof of concept, so for now we made a change so that the tcp server closes the connection. After this change I do not see the process hanging.
    The TCP server returns a string, but somehow the response string is not properly handled.
    I use below code and the context variable ends up an empty string.

    <xsl:variable name="response">
       <dp:url-open target="{$tcp-url}" response="binaryNode" data-type="xml">
          <xsl:value-of select="$commit" />
       </dp:url-open>
    </xsl:variable>
    <Conversion>
    <str>
    <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/>
    </str>
    </Conversion>
     
    <xsl:variable name="text" select="/Conversion/str/text()" />
    <dp:set-variable name="'var://context/ECM/text'" value="string($text)" />
    
    Updated on 2014-03-25T03:31:22Z at 2014-03-25T03:31:22Z by iron-man
    • HermannSW
      HermannSW
      4369 Posts
      ACCEPTED ANSWER

      Re: url-open for TCP server response attribute

      ‏2011-05-07T00:47:08Z  in response to SystemAdmin
      > It turned out that the datapower is not closing the tcp connection. The client (datapower) opens the connection, so is the client
      > supposed to close the connection?
      You are right, the server has to close the TCP connection.
      For XML DataPower would know when the end of message is reached.
      For Non-XML the end of message indicator is the closing of the TCP connection.
      If the server does not close the connection DataPower does not know whether additional data might get sent or not.

      So dp:url-open waits until connection gets closed or timeout.
      You may set timeout attribute of dp:url-open.
      (see posting Adding latency during service development for backend simulation for an interesting application of the timeout value)

      > Anway, we have to finish this proof of concept, so for now we made a change so that the tcp server
      > closes the connection. After this change I do not see the process hanging.
      > ...
      Good.

       
      Hermann<myXsltBlog/>
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-06T21:31:54Z  in response to SystemAdmin
    I found that select="dp:decode(dp:binary-encode($response/result/binary),'base-64')" works but accessing Conversion/str does not.
    So I just use the decode(binary-encode) value.

    <!-- binary-encode and decode works but accessing Conversion/str does  not work -->
    <Conversion>
    <str>
    <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/>
    </str>
    </Conversion>
     
    <!-- below line does not work and text will end up an empty string -->
    <xsl:variable name="text" select="/Conversion/str/text()" />
    <dp:set-variable name="'var://context/ECM/text'" value="string($text)" />
     
    <!-- use below -->
    <xsl:variable name="responseString">
    <xsl:value-of select="dp:decode(dp:binary-encode($response/result/binary),'base-64')"/>
    </xsl:variable>
     
    <dp:set-variable name="'var://context/ECM/responseString'" value="string($responseString)" />
    
    Updated on 2014-03-25T03:31:17Z at 2014-03-25T03:31:17Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-09T19:46:07Z  in response to SystemAdmin
    Please confirm that timeout attribute to url-open works.
    I tested the timeout attribute by generating a datapower hanging condition
    (use <xsl:value-of select="'xxx'" instead of <![CDATA xxx ]>)
    The timeout is set to 5 sec.
    With the modified commit variable, TCP server does not close the connection.
    In 5 sec after the connection is established, the datapower is supposed to close the connection.
    But it does not.
    Would you please test the timeout attribute in your environment, and let me know the result

    <xsl:variable name="commit">
    <commit><xsl:value-of select="'0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0'" />
    </commit>
    </xsl:variable>
     
    <xsl:variable name="response">
    <dp:url-open target="{$tcp-url}" response="binaryNode" timeout="5" data-type="xml">
    <xsl:value-of select="$commit" />
    </dp:url-open>
    </xsl:variable>
    
    Updated on 2014-03-25T03:31:07Z at 2014-03-25T03:31:07Z by iron-man
    • HermannSW
      HermannSW
      4369 Posts
      ACCEPTED ANSWER

      Re: url-open for TCP server response attribute

      ‏2011-05-10T01:15:02Z  in response to SystemAdmin
      Hello,

      there is no big difference between your previous version with CDATA and <xsl:value-of ...>.

      But you are right, the timeout attribute of <dp:url-open ...> does not work for tcp://... connections.

      I tested that with just starting netcat listening on port 8080, without any echo command before.
      In this case netcat displays the (commit) string received from DataPower and DataPower waits for input entered in the shell where "nc -l 8080" was started.
      But DataPower waits and waits and does not timeout after 5 seconds.

      Please see this posting on a working timeout scenario:
      https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/entry/adding_latency_during_service_development_for_backend_simulation94

      The difference was using the HTTP protocol instead of tcp://.

      Now I changed your stylesheet to use the HTTP protocol against a sleep CGI script on a webserver:
      
      $ cat /var/www/cgi-bin/sleep.pl #!/usr/bin/perl   print 
      "echo Content-Type: text/plain\n\n"; print 
      "Hello ..."; sleep(15); print 
      "... world!\n"; $
      


      Here is the diff of your stylesheet to the previous version:
      
      $ diff tcp3.xsl tcp4.xsl 8c8 < <xsl:variable name=
      "tcp-url" select=
      "'tcp://dp4-l3.boeblingen.de.ibm.com:8080'" /> --- > <xsl:variable name=
      "tcp-url" select=
      "'http://dp4-l3.boeblingen.de.ibm.com/cgi-bin/sleep.pl'" /> 10c10 < <commit><![CDATA[0&DocClass=CMBS&xxFileDirxx=/MSAR/cmbs_store/&xxFileNamexx=test&xxFileExtxx=txt&FormatType=TXT&xxNumPgxx=0]]> --- > <commit><xsl:value-of select=
      "'0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0'" /> 15c15 < <dp:url-open target=
      "{$tcp-url}" response=
      "binaryNode" data-type=
      "xml"> --- > <dp:url-open target=
      "{$tcp-url}" response=
      "binaryNode" timeout=
      "5" data-type=
      "xml"> $
      


      Running this against the sleep service times out after 5 seconds:
      
      $ time java coproc2 tcp4.xsl some.xml http:
      //dp3-l3.boeblingen.de.ibm.com:2223   real    0m5.401s user    0m0.360s sys     0m0.122s $
      


      And changing the sleep value from 15 to 1 works as expected:
      
      $ time java coproc2 tcp4.xsl some.xml http:
      //dp3-l3.boeblingen.de.ibm.com:2223 Hello ...... world!   real    0m1.374s user    0m0.331s sys     0m0.115s $
      


      Please create a PMR on timeout attribute of <dp:url-open ...> not working for tcp://.

      Either
      • this will be fixed (likely)
      • or at least the documentation will point out that timeout does not work for tcp:// (and why)

       
      Hermann<myXsltBlog/>
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-10T12:00:41Z  in response to SystemAdmin
    Thank you for testing and confirming that the timeout attribute works with url-open for http, but not with the one for tcp.
    I'm using 3.8.0.12, which is the latest under the 3.8.0.x branch released in late April 2011, so I think this is a bug.
    I've also noticed this. After datapower hangs and I kill the process by cntl-C, datapower stops writing transactions in the probe.
    This occurs not only for this transaction but also all subsequent transactions.
    And it impacts all domains on the same device.
    To recover from this, domain restart does not work and device restart is needed.
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-10T12:53:35Z  in response to SystemAdmin
    Question about closing connection:

    What will happen if network problem arises right after tcp communication is established from datapower to the TCP server, and datapower does not have the timeout set.

    (1) Datapower will hang forever becasue TCP server cannot send a close connection signal back to the datapower?

    (2) Underlining os will send network error back to the datapower, and the datapower will not hang?

    (3) Datapower will be aware of the network problem somehow and it will not hang?

    Please let me know.
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-10T18:59:11Z  in response to SystemAdmin
    I opened a PMR: 59398 499 000-timeout attribute for tcp url-open does not work.
    • HermannSW
      HermannSW
      4369 Posts
      ACCEPTED ANSWER

      Re: url-open for TCP server response attribute

      ‏2011-05-12T18:34:08Z  in response to SystemAdmin
      > I opened a PMR: 59398 499 000-timeout attribute for tcp url-open does not work.
      >
      Thank you for doing that -- it is really needed.

      Regarding your questions, I started two requests and waited and waited ... and finally stopped them after half an hour.
      Both showed up as established in "Status-->IP-Network-->TCP Port Status" the whole time.

       
      Now let me take the opportunity to riddle you/the other readers of this DataPower Forum:
      Based on the information in this thread there exists a (simple) workaround
      on DataPower side to use <dp:url-open ...> in a stylesheet and access a
      rawTCP backend and get a timeout before PMR 59398,499,000 gets fixed.

      This shows the workaround in action with a timeout of 8 seconds:
      $ time java coproc2 tcp5.xsl some.xml http://dp3-l3.boeblingen.de.ibm.com:2223
       
      real    0m8.313s
      user    0m0.374s
      sys     0m0.103s
      $
      


      Have fun ;-)

       
      Hermann <myXsltBlog/>
      Updated on 2014-03-25T03:30:34Z at 2014-03-25T03:30:34Z by iron-man
      • Liv2luv
        Liv2luv
        573 Posts
        ACCEPTED ANSWER

        Re: url-open for TCP server response attribute

        ‏2011-05-12T19:36:17Z  in response to HermannSW
        Just a guess, will any one of these two time out parameters make the trick ?

        var://local/_extension/timeout
        var://service/mpgw/backend-timeout

        Thanks.
        • HermannSW
          HermannSW
          4369 Posts
          ACCEPTED ANSWER

          Re: url-open for TCP server response attribute

          ‏2011-05-12T21:12:09Z  in response to Liv2luv
          Hi Suresh,

          > Just a guess, will any one of these two time out parameters make the trick ?
          >
          > var://local/_extension/timeout
          > var://service/mpgw/backend-timeout
          >
          No, the PMR was created because
          <dp:url-open target="'tcp://...'" ...>...</dp:url-open>
          

          hangs indefinitely when TCP server does not close the connection, regardless what timeout values get specified.

           
          Hermann <myXsltBlog/>
          Updated on 2014-03-25T03:30:29Z at 2014-03-25T03:30:29Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-12T20:26:37Z  in response to SystemAdmin
    Please post your tcp5.xsl.

    The TCP call using url-open is a side call and I use loopbackFirewall as the main backend
    (The loopback firewall is set to 'var://service/routing-url')

    The previous responder is mentioning the service's backned value:

    Q1: Is he correct?
    Q2: If correct, does it work for the side call or just for the main backend call?

    • HermannSW
      HermannSW
      4369 Posts
      ACCEPTED ANSWER

      Re: url-open for TCP server response attribute

      ‏2011-05-12T21:45:16Z  in response to SystemAdmin
      > Please post your tcp5.xsl.
      >
      I will post, but later after others had a chance to provide a solution.

      I responded the solution back to you in the PMR and by email because this is urgent to you.

      > The TCP call using url-open is a side call and I use loopbackFirewall as the main backend
      > (The loopback firewall is set to 'var://service/routing-url')
      >
      > The previous responder is mentioning the service's backned value:
      >
      > Q1: Is he correct?
      No.

      > Q2: If correct, does it work for the side call or just for the main backend call?
      >
      Hermann<myXsltBlog/>
  • HermannSW
    HermannSW
    4369 Posts
    ACCEPTED ANSWER

    Re: url-open for TCP server response attribute

    ‏2011-05-16T22:44:50Z  in response to SystemAdmin
    OK, no new ideas for the workaround.

    Let me give a hint:
    Making use of
    <dp:url-open target="http://..." response="binaryNode" timeout="5">
    

    with working timeout (see above) is the basis of the workaround.
    How can the access to remote TCP server (tcp://...) be done based on this?

     
    Hermann <myXsltBlog/>
    Updated on 2014-03-25T03:30:06Z at 2014-03-25T03:30:06Z by iron-man
    • SystemAdmin
      SystemAdmin
      6772 Posts
      ACCEPTED ANSWER

      Re: url-open for TCP server response attribute

      ‏2011-05-17T13:52:05Z  in response to HermannSW
      Hermann

      I will be testing this very problem in a couple of weeks. So this added 'timeout' attribute actually does end the tcp connection? If so, great news. This particular issue has been causing the connections to build up, and hang the boxes eventually. I automated some things for the client at silent hours, which solved the issue, like firmware reloads (crazy, yes, but fairly quick and painless). My thinking was to utilize a stateful raw tcp handler. Currently they are using an ssl proxy to make these transactions, which is not preferable. It's basically TLS over TCP, and the connections build, stay established, and then the box hangs/and/or reboots as I understand it. But if this timeout actually ends the connection, then I will definitely go that route first, vs. the stateful raw handler.

      Thanks for the feedback -
      • HermannSW
        HermannSW
        4369 Posts
        ACCEPTED ANSWER

        Re: url-open for TCP server response attribute

        ‏2011-05-23T13:47:10Z  in response to SystemAdmin
        Kirby,
        >
        > I will be testing this very problem in a couple of weeks. So this added 'timeout' attribute actually does end the tcp connection?
        > If so, great news. This particular issue has been causing the connections to build up, and hang the boxes eventually. I automated
        > some things for the client at silent hours, which solved the issue, like firmware reloads (crazy, yes, but fairly quick and
        > painless). My thinking was to utilize a stateful raw tcp handler. Currently they are using an ssl proxy to make these transactions,
        > which is not preferable. It's basically TLS over TCP, and the connections build, stay established, and then the box hangs/and/or
        > reboots as I understand it. But if this timeout actually ends the connection, then I will definitely go that route first, vs. the
        > stateful raw handler.
        >

        the timeout attribute should work for "tcp:///..." connections (for dp:url-open() as well as as static backend of a MPGW).
        But it does not until (new) PMR 59398,499,000 (IC76418) has been fixed.

        I do not completely understand your SSL proxy problem.
        Perhaps best would be if you create a new thread stating the problem first and then your ideas on how to workaround the problem with tcp:///.

         
        Hermann<myXsltBlog/>
        • HermannSW
          HermannSW
          4369 Posts
          ACCEPTED ANSWER

          Re: url-open for TCP server response attribute

          ‏2011-05-23T14:49:36Z  in response to HermannSW
          OK,

          no further submission after giving the hint, and I have been asked to provide the solution.

           
          As stated in the hint the workaround makes use of the fact that timeout attribute works for http protocol, see tcp4.xsl in this posting:
          https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14620016#14614888
          For the riddle
          https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14620016#14616340

          this is small diff between tcp4.xsl and tcp5.xsl:
          
          $ diff tcp4.xsl tcp5.xsl 8c8 < <xsl:variable name=
          "tcp-url" select=
          "'http://dp4-l3.boeblingen.de.ibm.com/cgi-bin/sleep.pl'" /> --- > <xsl:variable name=
          "tcp-url" select=
          "'http://127.0.0.1:8081'" /> 15c15 < <dp:url-open target=
          "{$tcp-url}" response=
          "binaryNode" timeout=
          "5" data-type=
          "xml"> --- > <dp:url-open target=
          "{$tcp-url}" response=
          "binaryNode" timeout=
          "8" data-type=
          "xml"> $
          


          Just go to an internal HTTP service listening on port 8081.

          The last piece needed for the workaround now is a HTTP2rawTCP service on port 8081.

          In posting "Processing of Non-XML rawTCP data is possible on DataPower appliances"
          http://www.ibm.com/developerworks/forums/thread.jspa?threadID=364522

          a rawTCP2HTTP service was provided.
          Now this is how you can setup a HTTP2rawTCP service:
          • create a MPGW
          • HTTP FSH listening on 127.0.0.1:8081
          • pass-thru request type
          • pass-thru response type
          • "default" policy
          • static backend type
          • backend URL "tcp://dp4-l3.boeblingen.de.ibm.com:8081"

          That's it -- this service just
          • takes HTTP request
          • sends rawTCP to backend
          • receives rawTCP response from backend
          • returns HTTP response

          Because of the timeout set for the (local) http-connection the workaround is complete.
          (see Figure 7 from Multi-Protocol Gateway configuration)

           
          Hermann<myXsltBlog/>
          • Liv2luv
            Liv2luv
            573 Posts
            ACCEPTED ANSWER

            Re: url-open for TCP server response attribute

            ‏2011-05-23T19:57:33Z  in response to HermannSW
            Thank you for the solution. It was interesting.

            So, to implement this workaround, we need to have service chaining:

            MPG-A (http) <-> (http) MPG-B (tcp) <-> TCP Server

            In this pattern, When the connection to TCP server is broken after the connection is established (MPG-B to TCP), datapower connection would still hang in MPG-B ? Please validate.

            Thanks.
            • HermannSW
              HermannSW
              4369 Posts
              ACCEPTED ANSWER

              Re: url-open for TCP server response attribute

              ‏2011-05-23T21:38:20Z  in response to Liv2luv
              Hi Suresh,

              > Thank you for the solution. It was interesting.
              >
              > So, to implement this workaround, we need to have service chaining:
              >
              > MPG-A (http) <-> (http) MPG-B (tcp) <-> TCP Server
              >
              > In this pattern, When the connection to TCP server is broken after the connection is established (MPG-B to TCP), datapower connection would still hang in MPG-B ? Please validate.
              >
              thank you for this very good question.

              I just tried it and this time I prefixed the netcat command with time command:
              
              [stammw@dp4-l3 ~]$ time nc -l 8081 0|DocClass=CMBS|xxFileDirxx=/MSAR/cmbs_store/|xxFileNamexx=test|xxFileExtxx=txt|FormatType=TXT|xxNumPgxx=0 real    2m18.165s user    0m0.000s sys     0m0.000s [stammw@dp4-l3 ~]$
              


              I just sent a single request against this "broken" TCP service:
              
              $ time java coproc2 tcp5.xsl some.xml http:
              //dp3-l3.boeblingen.de.ibm.com:2223   real    0m8.588s user    0m0.338s sys     0m0.058s $
              


              After netcat timeout I did this experiment again, but this time I took a packet capture on DataPower.
              Exactly after 120 seconds DataPower did send a [FIN, ACK] to the TCP server.

              So PMR 59398,499,000 (IC76418) is just about timeout being always 120 seconds instead of being configurable.

              Anyone making use of this solution now before IC76418 is fixed needs to make sure by SLM configuration
              that at most 500 requests per second are accepted for such a TCP service in order to not run out of free ports
              before the waiting connections timeout after 2 minutes.

               
              Hermann<myXsltBlog/>
              • HermannSW
                HermannSW
                4369 Posts
                ACCEPTED ANSWER

                Re: url-open for TCP server response attribute

                ‏2011-05-24T08:25:12Z  in response to HermannSW
                During my morning run to the IBM lab I thought about what I wrote last night.

                > ...
                > After netcat timeout I did this experiment again, but this time I took a packet capture on DataPower.
                > Exactly after 120 seconds DataPower did send a [FIN, ACK] to the TCP server.
                >
                The reason for terminating the connection after 2 minutes was the Back Side Timeout of the workaround MPGW.

                > So PMR 59398,499,000 (IC76418) is just about timeout being always 120 seconds instead of being configurable.
                >
                This is not correct, as stated in this posting above tcp:/// connections hang indefinitely for dp:url-open.
                That is what the PMR is about.

                > Anyone making use of this solution now before IC76418 is fixed needs to make sure by SLM configuration
                > that at most 500 requests per second are accepted for such a TCP service in order to not run out of free ports
                > before the waiting connections timeout after 2 minutes.
                >
                Now this can be handled much better as I found out during above mentioned run.
                Just set Back Side Timeout of workaround MPGW to eg. 9 seconds in case of dp:url-open timeout of 8 seconds to local workaround MPGW.
                This way the dp:url-open will timeout after 8 seconds, and the connection from workaround MPGW to TCP server times out after 9 seconds!
                I tested it and it really works that way.
                This way SLM only needs to restrict access to the service at 7000 requests per second to prevent running out of free ports.

                Here is a little ASCII art representation of the whole dribble-workaround solution:
                
                .                              MPGW toRawTCPwithTimeout +---------------------------------------------------------+ !                                                         ! !    /-----------------------------------------------\    ! !    ! ...                                           !    ! !    ! <dp:url-open target=
                "'http://127.0.0.1:8081'" !    ! !    !   response=
                "binaryNode" timeout=
                "8">          !    ! !    !   ...                                         !    ! !    ! </dp:url-open>                                !    ! !    ! ...                                           !    ! !    \-----------------------------------------------/    ! !                                       !                 ! !                                       !                 ! !  +---------------------------------------------------+  ! !  !                                   NULL            !  ! !  !                                                   !  !      /  ------> !  !  Match( Url = * ) --- xform( local:/​/​/tcp5.xsl )  !  ! ----/---> !  !                                                   !  !    /  !  !                                   NULL            !  ! !  +---------------------------------------------------+  ! !                                                         ! !                                                         ! !                                                         ! <------ !                        (pass-thru)                      ! !                                                         ! +---------------------------------------------------------+         MPGW workaround +---------------+ HTTP FSH     !               !  tcp:
                //dp4-l3.boeblingen.de.ibm.com:8081 ---------------> !  (pass-thru)  ! ----------------------------------------> 127.0.0.1:8081  !               !        ( Back Side Timeout = 9 ) !               ! <--------------- !  (pass-thru)  ! <---------------------------------------- !               ! +---------------+
                

                Hermann&lt;myXsltBlog/>
                • SystemAdmin
                  SystemAdmin
                  6772 Posts
                  ACCEPTED ANSWER

                  Re: url-open for TCP server response attribute

                  ‏2011-06-06T21:50:23Z  in response to HermannSW
                  I am getting this error when using tcp:// on a static back end mpgw:

                  MNI_MPGW): URL 'tcp://10.50.65.21:9997/' can only be used for sending data
                  


                  I'm following the guide in this post, but I've got something incorrect. I have one mpgw with a dynamic backend via xsl transform action/stylesheet, using url-open to call the HTTP FSH of another MPGW. This first client request will be coming from a phone, but for now, I'm hitting it with firefox, with some phone/wap/xhtml add-ons. I did have my Transform Action to NULL but it is now set to Perhaps my xslt is off, since there is no 'data' in the call?

                  The 2nd MPGW is where the HTTP FSH receives the request and uses the tcp:// backend set statically. Pass-thru on both sides.

                  This same connection/phone application is working on Datapower with an SSL Proxy Service but has the problem of hanging TCP connections, thus, it is the timeout function that will ultimately solve the issue I think, so that's where this MPGW to MPGW workaround comes into play. Thank you for the information Hermann -

                  Kirby
                  Updated on 2014-03-25T03:29:12Z at 2014-03-25T03:29:12Z by iron-man
                  • Liv2luv
                    Liv2luv
                    573 Posts
                    ACCEPTED ANSWER

                    Re: url-open for TCP server response attribute

                    ‏2011-06-06T23:30:55Z  in response to SystemAdmin
                    > The 2nd MPGW is where the HTTP FSH receives the request and uses the tcp:// backend set statically. Pass-thru on both sides.

                    If you're using the second MPGW only to implement time-out feature; why not consider this instead so that you can implement variable timeout based on the message backend.

                    <xsl:choose>
                                            <xsl:when test="$tcpMessage">
                                                    <dp:set-variable name="'var://service/mpgw/backend-timeout'" value="'5'"/>
                                                    <dp:url-open target="{$tcpTarget}" response="binaryNode">
                                                            <xsl:copy-of select="$tcpMessage"/>
                                                    </dp:url-open>
                                            </xsl:when>
                                            <xsl:when test="$otherMessage">
                                                <dp:set-variable name="'var://service/mpgw/backend-timeout'" value="'10'"/>
                                                <dp:url-open target="{httpTarget}" ...>
                                                   <xsl:copy-of select="$httpMessage"/>       
                                                </dp:url-open>
                                            </xsl:when>
                                    </xsl:choose>
                    


                    Thanks.
                    Updated on 2014-03-25T03:29:07Z at 2014-03-25T03:29:07Z by iron-man
                    • SystemAdmin
                      SystemAdmin
                      6772 Posts
                      ACCEPTED ANSWER

                      Re: url-open for TCP server response attribute

                      ‏2011-06-07T13:36:06Z  in response to Liv2luv
                      Thanks Suresh

                      I will do that. I know this is a TCP connection using TLS only (still have to pin down details on that). Interestingly, when I use a backend of "tcps" or "tcpssl" I get "failed backside connection" When I use "tcp:" as a backend I am currently getting the "can only be used to send data" error I mentioned earlier. Possibly this is just that I need to test from an actual phone instead of a browser. There is also the option of putting the MPGW in front of the working SSL Proxy Service, and using the timeout function there, but in that example, there is nothing telling the SSL Proxy to use TCP - apparently it is just working, but builds up "Established" TCP connections after a couple of days, thus the box freezes or reboots itself - more details I need to gather today. I'll test out your XSL, along with some other things, and see what I get.

                      Kirby
                      • SystemAdmin
                        SystemAdmin
                        6772 Posts
                        ACCEPTED ANSWER

                        Re: url-open for TCP server response attribute

                        ‏2011-06-10T12:41:28Z  in response to SystemAdmin
                        Starting a new post - this is looking to be a separate issue regarding TCP connections on the front and back of Datapower.