CICS IPIC high availability

The IPIC high availability (HA) capability is supported for IPIC connections between IBM z/OS Connect servers and CICS® regions.

This capability ensures resilience of access to CICS and balances connections across a cluster of CICS regions. z/OS Communications Server offers two balancing mechanisms, TCP/IP port sharing and Sysplex Distributor. You can configure Sysplex Distributor to prefer local connections, for more information, see Sysplex distributor.
Note:
  1. IPIC high availability can be achieved by using either IPIC HA connections, as described below, or IPIC workload distribution, see CICS IPIC workload distribution.
  2. All IPIC CICS connections are persistent. A connection remains established until either it is broken, or if it is an HA connection, it is ended as a result of HA reconnection.

Establishment of an HA connection

Figure 1. IPIC HA connection flows to the generic and specific ports.
This diagram shows the flows in the establishment of an IPIC HA connection.

The CICS regions listen on two end points that are defined by TCPIPSERVICE resources. The generic end point is shared by all regions in the cluster and a specific end point that is used exclusively by a specific region. The IPCONN can be auto-installed, or pre-defined. When using security, make sure that the generic and specific TCPIPServices specify the same security credentials.

An IBM z/OS Connect server connects to a CICS region in the cluster by using the generic port specified in the zosconnect_cicsIpicConnection definition. The connection request on the shared generic port is intercepted by the connection-balancing mechanism and routed to one of the CICS regions in the cluster. The selected CICS region then returns the IP address and port of its specific end point to the IBM z/OS Connect server, and the connection is re-established using those.

HA connection failover

When an IPIC HA connection is lost, or closed because there is no heartbeat response, IBM z/OS Connect automatically attempts to establish a new connection by using the shared generic port. This provides a connection failover mechanism in the event that a CICS region becomes unresponsive or fails. The heartbeatInterval attribute on the zosconnect_cicsIpicConnection element sets the time that the connection must be inactive before a heartbeat request is sent to the CICS region. If no response is received from CICS within the interval time, IBM z/OS Connect closes the connection.

HA reconnection

From V3.0.56, z/OS Connect provides a mechanism to automatically reconnect an IPIC HA connection to enable re-balancing of connections. Connections can become unbalanced due to connection failover. A new connection is established before the existing connection is closed, and outstanding requests are allowed to complete. This results in an existing connection being switched to another CICS region while a workload is flowing, without request failures due to the switch process.

This capability is enabled by specifying a reconnect interval by using the reconnectInterval attribute on the zosconnect_cicsIpicConnection element. Optionally a preferred CICS region can be specified using the preferredSpecificHost and preferredSpecificPort attributes. Parameter settings for IPIC HA reconnection describes the reconnection behavior dependent on the settings of these attributes.

When the reconnect interval expires for an established connection:
  1. If a preference is specified and this matches the existing connection, z/OS Connect waits for another reconnect interval.
  2. If the existing connection does not match the preference, or no preference is specified, an attempt is made to establish a new IPIC connection, by using the shared port. The port sharing or Sysplex Distributor algorithm selects the best endpoint to keep connections balanced.
  3. If the connection succeeds, a preference is specified, and the newly connected CICS region does not match the preference, then the connection on the shared port is closed and z/OS Connect waits for another reconnect interval.
  4. If the newly connected CICS region matches the preference, or no preference is specified, then the connection on the shared port is closed and an attempt is made to establish the connection on the specific port.
  5. New requests are sent over the new connection.
  6. The previous connection is closed when all outstanding requests have been completed.
Note:
  • The preferred specific host and port values are not used to directly establish a connection. They are used only to decide whether a connection should be switched. This ensures that the port sharing technology remains in control of connection balancing.
  • Updates to the preferredSpecificHost and preferredSpecificPort attributes take immediate effect on existing connections, whereas updates to the reconnectInterval attribute take effect only for new connections.
Parameter settings for IPIC HA reconnection
The reconnection behavior is controlled by the specification of the reconnectInterval, preferredSpecificHost and preferredSpecificPort settings. Choose one of the following options, according to the desired reconnection behavior.

reconnectInterval=0
Reconnection is disabled. This is the default.

reconnectInterval>0
preferredSpecificHost=<Host name of preferred LPAR>. See Tip.
preferredSpecificPort=<Port number of preferred CICS region>
Reconnection is attempted when the time interval expires and the existing connection is not using the preferred host and port. Use this option when using Sysplex Distributor and a particular CICS region is preferred.

reconnectInterval>0
preferredSpecificHost=<Host name of preferred LPAR>. See Tip.
Reconnection is attempted when the time interval expires and the existing connection is not to the preferred host. Use this option when using Sysplex Distributor and a particular LPAR is preferred.

reconnectInterval>0
preferredSpecificPort=<Port number of preferred CICS region>
Reconnection is attempted when the time interval expires and the existing connection is not using the preferred port. Use this option when using port sharing and a particular CICS region is preferred.

reconnectInterval>0
Reconnection is attempted when the time interval expires. Use this option to enable regular connection re-balancing.

Tip: From V3.0.57.0, if the preferred CICS region is on the same LPAR as the z/OS Connect server, you can specify preferredSpecificHost="local". For more information, see Setting preferredSpecificHost="local".
Check the messages.log file to see when a reconnection occurs. For example, if preferredSpecificHost="192.0.2.1", the following log entries describe the reconnection process:
  1. An initial connection is established: 
    BAQR0680I: CICS connection cicsConn established to host 192.0.2.1 on port 1234 with 100 sessions.
  2. The connection fails: 
    BAQR0664E: Communication failure for CICS connection cicsConn to host 192.0.2.1 on port 1234, reason: java.io.IOException: End of stream.
  3. A connection is established to a non-preferred CICS region: 
    BAQR0680I: CICS connection cicsConn established to host 192.0.2.2 on port 9876 with 100 sessions.
  4. HA reconnection establishes a connection to the preferred CICS region: 
    BAQR0704I: CICS connection cicsConn established to host 192.0.2.1 on port 1234 with 100 sessions, due to reconnection.
  5. The connection to the non-preferred CICS region is closed: 
    BAQR0705I: CICS connection cicsConn to host 192.0.2.2 on port 9876 has closed, due to reconnection.
Setting preferredSpecificHost="local"
From V3.0.57.0, if the preferred CICS region is on the same LPAR as the z/OS Connect server, you can specify preferredSpecificHost="local". A message like the following is then written to messages.log listing the local IP addresses detected:
BAQR0706I: A preferred host of local is configured for CICS connection cicsConn, local IP addresses that are detected for reconnection are [192.0.2.1, 192.0.2.2].

To use this setting, you must ensure that for each CICS region that is listening on the shared port, the specific IP address that is used for the HA connection is not a DVIPA IP address. All DVIPA IP addresses appear to be local to z/OS Connect. The specific IP address that is used by CICS is that specified in the HOST attribute of the TCPIPSERVICE definition. If HOST is specified as ANY, then CICS uses the default local host. The default local host is either that specified by the PRIMARYINTERFACE statement in the TCP/IP profile, if that is not specified, then it is the first address in the HOME list, and if no HOME statements are defined, then it is the first IPv4 INTERFACE statement that is listed in the profile.

To determine the default local host, issue the netstat -h command to see which interface has the Primary flag.

For further information, see Configuring IPIC High Availability.