Topic
  • 4 replies
  • Latest Post - ‏2016-11-04T14:29:44Z by vickas
vickas
vickas
10 Posts

Pinned topic TCPIP Client Input Node not picking the messages sometimes after recieving close connection from server

‏2016-10-25T06:04:30Z |

Hie All ,
IIB version - 9.0.0.2  - Active - passive Setup
MQ version - 7.5 
OS - RHEL 6.5 

we have a flow with TCP Client Input & Output nodes wihich is able to exchange the data with the server . This flow is on Prod & all looks fine except v.v.v few times . 

The problem comes when we get a close connection from server .
The flow reconnects to the server but after this the TCP client Input node fails to pick any message ( we see the messages being recieved onto the ESB server by verifying the TCP dump ) and at the same time the the flow is able to send the messages to the server. 

This problem happens only sometimes . 
we see the below exception when connection gets closed by the server : 

"RecoverableException:  exception in write Broken pipe "


can some experts tell any work around or the parameter that has to be changed . we are not using any configurable service .
server disconnects the flow because they have a logic to close the connection when they do not recieve ACK msgs from the client within some time period . 
It is at some fixed time , we have a BATCH message to be processed and while processing we see TCPIP Input node picking the messages at very slow rate ( observed this from application logs ) even though the messages reach onto Esb server on time ( observed from TCP dump ) & not able to send the ACKs on time. 
can you please tell how do we have to handle high volume of Data? 
which param on config service helps us ?



 

Updated on 2016-10-25T06:40:52Z at 2016-10-25T06:40:52Z by vickas
  • fjb_saper
    fjb_saper
    159 Posts

    Re: TCPIP Client Input Node not picking the messages sometimes after recieving close connection from server

    ‏2016-11-04T13:00:58Z  

    what have you tried so far to scale up?

  • vickas
    vickas
    10 Posts

    Re: TCPIP Client Input Node not picking the messages sometimes after recieving close connection from server

    ‏2016-11-04T13:27:42Z  
    • fjb_saper
    • ‏2016-11-04T13:00:58Z

    what have you tried so far to scale up?

    hie fjb ,
            thanks for the reply .
              since  the flow is already on production we are  not allowed to do any changes  to the flow .
    The main challenge we see here is we do not see this exception on the TEST  server .
    As a part of our analysis , we enabled the  user trace , collected the netstat  status and TCP dump .

    we were able to find  out that even though connection  was there ( netstat says ESTABLISHED)  after the disconnection & reconnection ( closed by server ) the TCP input node was not able to pick the msgs ( this is major issue ) but at the same time on the connection that was established , TCP cleint outputnode was able to send the data ( verified by TCP dump).
    Thinking of enhancing  the  flow  with  configurable service  and set the RECV & SEND BUffer Size 1024 KB & set TCP No_
    DELAY  to TRUE & Close_UnprocessedData to TRUE .
    As far i understand ,  the current  flow without configurable services  uses its buffer from the OS which is 156KB  but what i dont understand is , why  the flow picks at low pace   ( this is another issue ) while processing the Batch msg when we see from TCP dump that the data is recieved on to ESB  as and then sent by the server system.
      when the flow has to send and pick the msgs  at the same time , will the tcp connection manager  prioritizes to send the packets  and then pick the msgs ?  will it wait for the packets to be sent & then pick the msgs ?
    Can you please give me your inputs  ?? settings those parameters  will lead to any other consequences ?
     

  • fjb_saper
    fjb_saper
    159 Posts

    Re: TCPIP Client Input Node not picking the messages sometimes after recieving close connection from server

    ‏2016-11-04T14:06:17Z  
    • vickas
    • ‏2016-11-04T13:27:42Z

    hie fjb ,
            thanks for the reply .
              since  the flow is already on production we are  not allowed to do any changes  to the flow .
    The main challenge we see here is we do not see this exception on the TEST  server .
    As a part of our analysis , we enabled the  user trace , collected the netstat  status and TCP dump .

    we were able to find  out that even though connection  was there ( netstat says ESTABLISHED)  after the disconnection & reconnection ( closed by server ) the TCP input node was not able to pick the msgs ( this is major issue ) but at the same time on the connection that was established , TCP cleint outputnode was able to send the data ( verified by TCP dump).
    Thinking of enhancing  the  flow  with  configurable service  and set the RECV & SEND BUffer Size 1024 KB & set TCP No_
    DELAY  to TRUE & Close_UnprocessedData to TRUE .
    As far i understand ,  the current  flow without configurable services  uses its buffer from the OS which is 156KB  but what i dont understand is , why  the flow picks at low pace   ( this is another issue ) while processing the Batch msg when we see from TCP dump that the data is recieved on to ESB  as and then sent by the server system.
      when the flow has to send and pick the msgs  at the same time , will the tcp connection manager  prioritizes to send the packets  and then pick the msgs ?  will it wait for the packets to be sent & then pick the msgs ?
    Can you please give me your inputs  ?? settings those parameters  will lead to any other consequences ?
     

    You have to make a huge distinction between online processing (sparse) and batch processing (dense).

    A number of facts will impact batch processing:

    • available CPU
    • available memory
    • paging
    • bottlenecks in IO, Disk, etc ...

    And this might become so severe that it will take considerably more time to process anything...

    With IIB9 and up you could throttle the batch using workload policies. This would impact the CPU usage first most probably...

  • vickas
    vickas
    10 Posts

    Re: TCPIP Client Input Node not picking the messages sometimes after recieving close connection from server

    ‏2016-11-04T14:29:44Z  
    • fjb_saper
    • ‏2016-11-04T14:06:17Z

    You have to make a huge distinction between online processing (sparse) and batch processing (dense).

    A number of facts will impact batch processing:

    • available CPU
    • available memory
    • paging
    • bottlenecks in IO, Disk, etc ...

    And this might become so severe that it will take considerably more time to process anything...

    With IIB9 and up you could throttle the batch using workload policies. This would impact the CPU usage first most probably...

    hi  fjb , 

              thanks for the reply .
    here Batch msg means around 2500 msgs  being triggered  by system onto queue. when i say 2500 messages ,it looks like a very small number but  the flow has to send all those 2500 msgs (once recieved onto queue)    continuously  to the server for which i recieve Acks & corresponding Data  msg for  which again an Ack has to be sent back by the flow to the server.
    We see this TCP Client Input node not able to pick the messages at great pace ( as and when recieved onto esb server )  when batch is being processed . 
    which parameter settings  can help us to pick the messages fastly ? We do not have any other alternative to break down the batch .
    how could it impact if we increase the RCV & SNF buffer size from default 156 to 1024 ?? 
    how do i have to avoid the msgs being piled up  on tcp buffer ?