Network considerations between CDC Replication datastores and Db2 for z/OS

The CDC Replication Engine for Db2® for z/OS® Remote Source requires long-running JDBC connections between the machine where CDC Replication is installed and the mainframe server where the remote capture component reads the log. You must configure your firewall to allow such long-running JDBC connections.

Each connection has two sides: On the server side is a Db2 thread called CHC-LOGREADER-subname or CHC-LOGREADER-SHAREDSCRAPE, and on the client side is a Java™ connection object that the CDC Replication datastore holds. The client side of the connection issues the data requests to CHC-LOGREADER to retrieve data. If a network problem occurs between the datastore and Db2 for z/OS, the client and server connection objects become isolated.

Server side considerations

The CDC Replication system parameter mirror_head_of_log_sleep_milliseconds controls how long CDC Replication sleeps when it reaches head of log. The default value is 500 milliseconds. During the sleep time, no client calls are made to the server side of the connection, and therefore CHC-LOGREADER is marked as IDLE by Db2. The CHC-LOGREADER can be terminated if the sleep time exceeds Db2's IDLE timeout settings. Depending on parameter settings, Db2 can terminate CHC-LOGREADER and remove it from Db2's list of threads, or issue an ABEND with error code 00D3003B because the connection was idle for too long. Therefore, you need to ensure that mirror_head_of_log_sleep_milliseconds is less than Db2's IDLE timeout settings.

Db2 z/OS handles IDLE threads differently depending on the parameter settings for IDTHTOIN and DSN_PROFILE_TABLE/ATTRIBUTES (which specifies how Db2 reacts on timeout for specific threads). Consult the Db2 for z/OS documentation for these values so that idle Db2 threads can be handled gracefully during network problems.

The TCPKPALV parameter can be set in Db2 to configure the TCP KeepAlive feature.

Db2's KeepAlive feature serves the following purposes:

It allows Db2 to detect when communication stops because of network problems.
It keeps traffic going to prevent firewalls from closing idle connections.

Client side considerations

It is recommended that you turn on the advanced JDBC parameter KeepAlive. When a network problem causes the RST or FIN packets for closing the connection to be lost, the connection cannot be closed without the aid of KeepAlive. In this case, the CDC Replication subscription keeps running while no data is replicated to the target. The KeepAlive setting enables the subscription to detect the network problem and stop gracefully with an error message.

The JDBC driver's KeepAlive feature serves the following purposes:

It allows connections to close when communication stops because of network problems.
It keeps traffic going to prevent firewalls from closing idle connections.

To enable the KeepAlive feature, you need to specify the advanced JDBC parameter and configure from the operating system.

Using the CDC Replication configuration tool, add the following parameter and value to the Advanced JDBC Parameters section:
```
KeepAlive=true
```
Add or modify the operating system KeepAlive parameters.
Attention: Changing KeepAlive parameters affects all applications that run on the operating system. Review your operating system documentation and testing plans before making the changes. Improper changes can impair your system.
- Configuring KeepAlive on Linux®
  Add or modify the following parameters in /etc/sysctl.conf. Use sysctl -p to reload the file so that new settings can take effect:
  
  net.ipv4.tcp_retries2
  
  This value tells the kernel how many times to retry before ending a live TCP connection. The default value is 15, which corresponds to 13-30 minutes depending on the retransmission timeout (RTO). This value is calculated based on CPU speed, so the timeout might vary. The minimum recommended value for Linux is 8.
  
  net.ipv4.tcp_keepalive_time
  
  The interval between the last data packet sent (simple ACKs are not considered data) and the first KeepAlive probe, measured in seconds. The default is 7200 seconds (two hours).
  
  net.ipv4.tcp_keepalive_intvl
  
  The interval between subsequent KeepAlive probes, regardless of what the connection has exchanged in the meantime, measured in seconds. The default is 75.
  
  net.ipv4.tcp_keepalive_probes
  
  The number of unacknowledged probes to send before considering the connection dead and notifying the application layer. The default is 9.
  
  The values depend on the characteristics of your network. Some suggested values are:
```
net.ipv4.tcp_retries2 = 8
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6
```
- Configuring KeepAlive on Windows
  Use Regedt32.exe to modify the system registry. Add or modify the following registry keys in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters. Reboot when finished so that new settings can take effect:
  
  TcpMaxDataRetransmissions: REG_DWORD
  
  Controls the number of times that TCP retransmits an individual data segment before it aborts the connection. The default is 5.
  
  KeepAliveInterval: REG_DWORD
  
  Interval between KeepAlive packets when one is dropped, measured in milliseconds. The default is 1000 (1 second).
  
  KeepAliveTime: REG_DWORD
  
  How often TCP attempts to verify the connection, measured in milliseconds. The default is 7200000 (2 hours).
  
  TcpMaxDataRetransmissions: REG_DWORD
  
  Number of retries TCP does for a packet before giving up. The default is 5.

What happens when a network problem occurs

On the server side (mainframe), when a network problem occurs, interaction ceases between the client and CHC-LOGREADER. Db2 marks CHC-LOGREADER IDLE because no request is issued from the client side. Db2 handles IDLE thread differently depending on the settings for the parameters IDTHTOIN, TCPKPALV, and DSN_PROFILE_TABLE/ATTRIBUTES. CHC-LOGREADER can either be terminated and removed from Db2's list of threads, or Db2 can issue ABEND with error code 00D3003B because the connection has been idle for too long.

On the client side, if you enabled KeepAlive and configured the TCP/IP parameters on the operating system, then the subscription detects the network problem and stops the subscription with the following error message.

[CDC][DB2 JDBC Driver]Connection timed out (Read failed)

The time between when a network problem occurs and when the subscription detects it is determined by the operating system's TCP/IP settings. For example, in the following Linux settings,

net.ipv4.tcp_retries2 = 8
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6

When a packet cannot be sent, the value of net.ipv4.tcp_retries2 is used to control the number of times that the packet is retransmitted. The tcp_retries2 value of 8 corresponds to about 2 minutes.

When no packets can be received, the other three values control how long to wait. The time to detect a network problem also depends on net.ipv4.tcp_keepalive_time + (net.ipv4.tcp_keepalive_intvl * net.ipv4.tcp_keepalive_probes), which is 2 minutes with the above values. With these settings, you can expect the subscription to stop appropriately 2 minutes after not receiving any packets.

If the subscription stops while mirroring and is marked "persistent" with mirror_auto_restart_interval_seconds set to a nonzero value, the subscription can automatically restart. If the subscription stops during refresh, you need to investigate the problem and restart it manually.

Note: The observed behavior can be different depending on which long-lived connection hits an error. When any of the network errors are hit often, you need to investigate the network problems and address them.