Firewall idle connection timeout causes nodes to lose communication during low traffic times on Linux

Problem

Summary

This article discusses an issue where a firewall configured with an idle connection timeout can close connections to local nodes and nodes in other data centers during low traffic intervals.

Cause

The TCP keepalive may not be appropriately configured for your environment. The default idle connection timeout is usually 60 minutes.

Solution

1. Run this command to get a list of available kernel variables:

sysctl -A | grep net.ipv4

The following variables configure keepalive support in Linux:

• net.ipv4.tcp_keepalive_time

  Time of connection inactivity after which the first keep-alive request is sent.

• net.ipv4.tcp_keepalive_probes

  The number of keep-alive requests retransmitted before the connection is considered broken.

• net.ipv4.tcp_keepalive_intvl

  The time interval between keep-alive probes.

2. Configure these settings as appropriate for your environment. For example, to configure TCP keepalive timeout to 60 seconds with 3 probes, 10 seconds gap between each:

sudo sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=10

This command sets keepalive to detect dead TCP connections after 90 seconds (60 + 10 + 10 + 10). Additional traffic is negligible so it is safe to persist in these settings.

Tip: Instead of changing the system TCP settings, you can prevent reset connections during streaming by tuning the streaming_keep_alive_period_in_secs setting in cassandra.yaml.