Topic
5 replies Latest Post - ‏2013-09-20T14:27:28Z by AnuragDubey
AnuragDubey
AnuragDubey
7 Posts
ACCEPTED ANSWER

Pinned topic TCP sink : max buffering capacity

‏2013-09-19T11:16:29Z |

HI,

 In one of our projects we are using TCP sink as a server, suppose no client is connected to the sink,  data coming to sink is buffered in sink operator and is flushed out once client is connected to the sink. Can someone help by shedding some light on how much is size for this buffering.

Regards,

Anurag

 


 

  • hnasgaard
    hnasgaard
    200 Posts
    ACCEPTED ANSWER

    Re: TCP sink : max buffering capacity

    ‏2013-09-19T15:31:22Z  in response to AnuragDubey

    The TCPSink does not buffer tuples.  If the retryFailedSends is false (the default) the tuples are dropped.  If true then the operator blocks until a connection is made and it successfully sends the tuple.

    • Kevin_Foster
      Kevin_Foster
      98 Posts
      ACCEPTED ANSWER

      Re: TCP sink : max buffering capacity

      ‏2013-09-19T19:56:25Z  in response to hnasgaard

      Anurag, to add to this:

      "If the retryFailedSends is ... true then the operator blocks until a connection is made and it successfully sends the tuple."

      You can view the state of the running job in the Streams instance view to see statistics on the number of input and output tuples for each operator, memory usage, plus there's a setting to show congestion. That should help you to identify the memory requirements across all of your PE's when nothing is pulling data from your sink.

      Useful information if your job spans multiple machines, or even just one.

      -Kevin

      • AnuragDubey
        AnuragDubey
        7 Posts
        ACCEPTED ANSWER

        Re: TCP sink : max buffering capacity

        ‏2013-09-20T04:15:16Z  in response to Kevin_Foster

        Thanks for answering the questions, but retryFailedSends parameter has not been set in the TCPsink operator, still buffering is being done in operator. For example we a have a beacon operator then a custom operator which maintains record sent by beacon in map and then a tcpsink operator. beacon is continuously sending tuples to custom operator which maintains the tuples in a map and then forwarding them one by one to tcpsink. If we connect a client to tcpsink after some tuples have been sent from beacon to custom operator, all the tuples are received by client from sink. Just wanted to understand if what is happening is correct or we are doing a mistake.

        -Anurag

        • hnasgaard
          hnasgaard
          200 Posts
          ACCEPTED ANSWER

          Re: TCP sink : max buffering capacity

          ‏2013-09-20T11:58:20Z  in response to AnuragDubey

          Here is a sample app that, if you start, then connect to with telnet, will not get the initial tuples. 

          composite Main {
            graph
              stream<rstring s> Beat = Beacon() {
                param period : .5;
                output Beat : s = (rstring)IterationCount();
              }
              stream<Beat> C = Custom(Beat) {
                logic onTuple Beat : {
                  println("Submitting a tuple");
                  submit(Beat, C);
                }
              }
              () as Out = TCPSink(C) {
                param role : server;
                      port : 20000u;
                      flush : 1u;
              }
          }

           

          How does your sample code differ from this?  Can you post the source?

          • AnuragDubey
            AnuragDubey
            7 Posts
            ACCEPTED ANSWER

            Re: TCP sink : max buffering capacity

            ‏2013-09-20T14:27:28Z  in response to hnasgaard

            Hi,

             Sorry for our mistake, we were using two tcpsink operators and the param retryFailedSends was set to true in one of the sink operators.  Thanks for your help in pointing our mistake.

            Regards,

            Anurag