Topic
  • 5 replies
  • Latest Post - ‏2013-09-20T14:27:28Z by AnuragDubey
AnuragDubey
AnuragDubey
7 Posts

Pinned topic TCP sink : max buffering capacity

‏2013-09-19T11:16:29Z |

HI,

 In one of our projects we are using TCP sink as a server, suppose no client is connected to the sink,  data coming to sink is buffered in sink operator and is flushed out once client is connected to the sink. Can someone help by shedding some light on how much is size for this buffering.

Regards,

Anurag

 


 

  • hnasgaard
    hnasgaard
    200 Posts
    ACCEPTED ANSWER

    Re: TCP sink : max buffering capacity

    ‏2013-09-19T15:31:22Z  

    The TCPSink does not buffer tuples.  If the retryFailedSends is false (the default) the tuples are dropped.  If true then the operator blocks until a connection is made and it successfully sends the tuple.

  • Kevin_Foster
    Kevin_Foster
    98 Posts
    ACCEPTED ANSWER

    Re: TCP sink : max buffering capacity

    ‏2013-09-19T19:56:25Z  
    • hnasgaard
    • ‏2013-09-19T15:31:22Z

    The TCPSink does not buffer tuples.  If the retryFailedSends is false (the default) the tuples are dropped.  If true then the operator blocks until a connection is made and it successfully sends the tuple.

    Anurag, to add to this:

    "If the retryFailedSends is ... true then the operator blocks until a connection is made and it successfully sends the tuple."

    You can view the state of the running job in the Streams instance view to see statistics on the number of input and output tuples for each operator, memory usage, plus there's a setting to show congestion. That should help you to identify the memory requirements across all of your PE's when nothing is pulling data from your sink.

    Useful information if your job spans multiple machines, or even just one.

    -Kevin

  • hnasgaard
    hnasgaard
    200 Posts
    ACCEPTED ANSWER

    Re: TCP sink : max buffering capacity

    ‏2013-09-20T11:58:20Z  

    Thanks for answering the questions, but retryFailedSends parameter has not been set in the TCPsink operator, still buffering is being done in operator. For example we a have a beacon operator then a custom operator which maintains record sent by beacon in map and then a tcpsink operator. beacon is continuously sending tuples to custom operator which maintains the tuples in a map and then forwarding them one by one to tcpsink. If we connect a client to tcpsink after some tuples have been sent from beacon to custom operator, all the tuples are received by client from sink. Just wanted to understand if what is happening is correct or we are doing a mistake.

    -Anurag

    Here is a sample app that, if you start, then connect to with telnet, will not get the initial tuples. 

    composite Main {
      graph
        stream<rstring s> Beat = Beacon() {
          param period : .5;
          output Beat : s = (rstring)IterationCount();
        }
        stream<Beat> C = Custom(Beat) {
          logic onTuple Beat : {
            println("Submitting a tuple");
            submit(Beat, C);
          }
        }
        () as Out = TCPSink(C) {
          param role : server;
                port : 20000u;
                flush : 1u;
        }
    }

     

    How does your sample code differ from this?  Can you post the source?

  • hnasgaard
    hnasgaard
    200 Posts

    Re: TCP sink : max buffering capacity

    ‏2013-09-19T15:31:22Z  

    The TCPSink does not buffer tuples.  If the retryFailedSends is false (the default) the tuples are dropped.  If true then the operator blocks until a connection is made and it successfully sends the tuple.

  • Kevin_Foster
    Kevin_Foster
    98 Posts

    Re: TCP sink : max buffering capacity

    ‏2013-09-19T19:56:25Z  
    • hnasgaard
    • ‏2013-09-19T15:31:22Z

    The TCPSink does not buffer tuples.  If the retryFailedSends is false (the default) the tuples are dropped.  If true then the operator blocks until a connection is made and it successfully sends the tuple.

    Anurag, to add to this:

    "If the retryFailedSends is ... true then the operator blocks until a connection is made and it successfully sends the tuple."

    You can view the state of the running job in the Streams instance view to see statistics on the number of input and output tuples for each operator, memory usage, plus there's a setting to show congestion. That should help you to identify the memory requirements across all of your PE's when nothing is pulling data from your sink.

    Useful information if your job spans multiple machines, or even just one.

    -Kevin

  • AnuragDubey
    AnuragDubey
    7 Posts

    Re: TCP sink : max buffering capacity

    ‏2013-09-20T04:15:16Z  

    Anurag, to add to this:

    "If the retryFailedSends is ... true then the operator blocks until a connection is made and it successfully sends the tuple."

    You can view the state of the running job in the Streams instance view to see statistics on the number of input and output tuples for each operator, memory usage, plus there's a setting to show congestion. That should help you to identify the memory requirements across all of your PE's when nothing is pulling data from your sink.

    Useful information if your job spans multiple machines, or even just one.

    -Kevin

    Thanks for answering the questions, but retryFailedSends parameter has not been set in the TCPsink operator, still buffering is being done in operator. For example we a have a beacon operator then a custom operator which maintains record sent by beacon in map and then a tcpsink operator. beacon is continuously sending tuples to custom operator which maintains the tuples in a map and then forwarding them one by one to tcpsink. If we connect a client to tcpsink after some tuples have been sent from beacon to custom operator, all the tuples are received by client from sink. Just wanted to understand if what is happening is correct or we are doing a mistake.

    -Anurag

  • hnasgaard
    hnasgaard
    200 Posts

    Re: TCP sink : max buffering capacity

    ‏2013-09-20T11:58:20Z  

    Thanks for answering the questions, but retryFailedSends parameter has not been set in the TCPsink operator, still buffering is being done in operator. For example we a have a beacon operator then a custom operator which maintains record sent by beacon in map and then a tcpsink operator. beacon is continuously sending tuples to custom operator which maintains the tuples in a map and then forwarding them one by one to tcpsink. If we connect a client to tcpsink after some tuples have been sent from beacon to custom operator, all the tuples are received by client from sink. Just wanted to understand if what is happening is correct or we are doing a mistake.

    -Anurag

    Here is a sample app that, if you start, then connect to with telnet, will not get the initial tuples. 

    composite Main {
      graph
        stream<rstring s> Beat = Beacon() {
          param period : .5;
          output Beat : s = (rstring)IterationCount();
        }
        stream<Beat> C = Custom(Beat) {
          logic onTuple Beat : {
            println("Submitting a tuple");
            submit(Beat, C);
          }
        }
        () as Out = TCPSink(C) {
          param role : server;
                port : 20000u;
                flush : 1u;
        }
    }

     

    How does your sample code differ from this?  Can you post the source?

  • AnuragDubey
    AnuragDubey
    7 Posts

    Re: TCP sink : max buffering capacity

    ‏2013-09-20T14:27:28Z  
    • hnasgaard
    • ‏2013-09-20T11:58:20Z

    Here is a sample app that, if you start, then connect to with telnet, will not get the initial tuples. 

    composite Main {
      graph
        stream<rstring s> Beat = Beacon() {
          param period : .5;
          output Beat : s = (rstring)IterationCount();
        }
        stream<Beat> C = Custom(Beat) {
          logic onTuple Beat : {
            println("Submitting a tuple");
            submit(Beat, C);
          }
        }
        () as Out = TCPSink(C) {
          param role : server;
                port : 20000u;
                flush : 1u;
        }
    }

     

    How does your sample code differ from this?  Can you post the source?

    Hi,

     Sorry for our mistake, we were using two tcpsink operators and the param retryFailedSends was set to true in one of the sink operators.  Thanks for your help in pointing our mistake.

    Regards,

    Anurag