z/OS SWSA in a nutshell

Body

z/OS® IP security supports Sysplex-Wide Security Associations (SWSA).

 

In a sysplex environment, SWSA distributes IPsec Security Associations (SAs) to all of the target stacks of distributed DVIPAs.

In addition to IPsec DVIPA distribution, SWSA manages recovery for SAs associated with DVIPAs that have moved to an alternate TCP/IP stack (DVIPA takeover).  For this recovery to take place, stacks that back up the DVIPA do not receive SA data from the distributor directly but have access to the SA data in the EZBDVIPA coupling facility (CF) structure.  SWSA DVIPA takeovers will be discussed later.

To enable Sysplex-Wide Security Associations (SWSA) on a stack that has IP security enabled, add the DVIPSEC parameter to the IPSEC statement block in the TCP/IP profile.

If you need to protect intra-sysplex traffic (both the client and distributor in the sysplex) with IPsec SAs, you might need to configure the DVLOCALFLTR parameter on the IPSEC statement block in the TCP/IP profile.

 

Sysplex distributor

 

With SWSA enabled, the distributor is responsible for negotiating an IPsec SA with a remote peer. Once this SA/VPN is created, copies of the SA, known as Shadow SAs, are sent to all target stacks that can potentially receive workload for the DVIPA. The shadow SAs enable the distribution of cryptography to the target stacks. Whenever a new SA is negotiated or refreshed and the SA is installed in the distributor stack, a copy (shadow) of the SA is sent within the sysplex to the target hosts.

 

In sysplex distribution, TCP traffic protected by an IPsec SA with a sysplex-distributed DVIPA endpoint can be distributed to target hosts.

 

IPsec cryptography for outbound traffic is performed on the target host, and then sent directly into the network without being routed through the distributor.

 

IPsec cryptography for inbound traffic is performed on the target host in most cases. Inbound traffic is received on the distributor before being routed to the target host. The distributor must decrypt a small number of bytes to determine the appropriate target. The encrypted packet is then forwarded to the target which fully decrypts it. There are several cases where full decryption occurs on the distributor stack. One of the cases where this occurs is when the AES-GCM combined algorithm is used. With this algorithm, it is not possible to decrypt a portion of the message to determine to which stack the message should be sent. So, the message is fully decrypted. The decrypted packet is then GRE encapsulated and sent to the target over the XCF link.

 

Sysplex distributor – Displaying negotiated tunnels and shadow tunnels

 

To display negotiated tunnel information on a distributor stack, issue:  ipsec -y display

 

To display SWSA shadow tunnel info on target stacks, issue:  ipsec -y display -s
 

Sysplex distributor - Tunnel activation

 

When the initial Security Association is created between the client and distributor, IKED messages EZD1779I and EZD1775I will be written. Here is an example of these messages:

 

EZD1779I IKE version 2.0 security association 1 for tunnel Yxxx has the following attributes - encapsulation : TRANSPORT encryption : AES_CBC integrity : HMAC_SHA2_256_128 lifetime : 21600 lifesize : NONE VpnLife : NONE PFS : GROUP21

EZD1775I IKE version 2.0 security association 1 for tunnel Yxxx created for protecting TCP(6) traffic between y.y.y.y ports ALL and z.z.z.z ports ALL

 

TRMD message EZD0818I will also be issued for the distributor stack, as shown in this example:

 

EZD0818I Tunnel added: timestamp vpnaction= VPNActionName tunnelID= Yxxx  AHSPI= 0  ESPSPI= 123456789

 

On the target LPARs, TRMD message EZD0818I will also be seen for the addition of the shadow tunnel. For example:

 

EZD0818I Tunnel added: timestamp vpnaction= VPNActionName tunnelID= Yxxx  AHSPI= 0  ESPSPI= 123456789

 

The SAID (e.g. Yxxx) will be the same in the IKED and TRMD messages.

 

IKED messages are written to the IKED syslogd LOCAL4 destination file.

TRMD messages are written to the TRMD syslogd LOCAL4 destination file.

 

Sysplex distributor – Tunnel deletion

 

IKED message EZD1776I and TRMD message EZD0812I will be issued during tunnel deletion. Here is an example of these messages for the distributor:

 

EZD1776I IKE version 2.0 security association 1 for tunnel Yxxx deleted

 

TRMD message EZD0812I will also be issued, as shown in this example:

 

EZD0812I Tunnel deleted: timestamp vpnaction= VPNActionName tunnelID= Yxxx AHSPI= 0  ESPSPI= 123456789

 

On the target, TRMD message EZD0812I will also be seen for the deletion of the shadow tunnel. For example:

 

EZD0812I Tunnel deleted: timestamp vpnaction= VPNActionName tunnelID= Yxxx AHSPI= 0  ESPSPI= 123456789

 

Note: Shadow tunnels cannot be deleted, either manually using ipsec command options, or indirectly by IKE delete payloads. A shadow tunnel is only deleted when the negotiated tunnel created by IKED for the sysplex distributor is deleted.
 

EZBDVIPA Coupling Facility (CF) structure

 

As mentioned above, SWSA requires the use of a coupling facility structure, EZBDVIPA. The EZBDVIPA structure stores shared information about phase 1 and phase 2 SAs whose endpoint is a Sysplex DVIPA. For each DVIPA with data protected by IPsec, information is stored about the negotiated SAs for that DVIPA. This data is used for DVIPA takeover (described below). For IKEv1 SAs, information is stored for both the phase 1 and phase 2 SA. For IKEv2 SAs, only phase 2 (i.e. Child SA) data is stored.

 

For each phase 2 SA whose endpoint is a DVIPA, there is also a sequence number entry in the EZBDVIPA structure. This sequence number is shared by one or more target stacks that are sending data protected by the SA. The sequence number is used for DVIPA distribution.

 

The data in the EZBDVIPA CF structure can be displayed with the 'D NET,STATS,TYPE=CFS,STRNAME=EZBDVIPA,LIST=ALL,SCOPE=All'  command. Below is example output:

 

D NET,STATS,TYPE=CFS,STRNAME=EZBDVIPA,LIST=ALL,SCOPE=ALL

IST097I DISPLAY ACCEPTED                                        

IST350I DISPLAY TYPE = STATS,TYPE=CFS                          

IST1370I NETA.SSCP1A IS CONNECTED TO STRUCTURE EZBDVIPA                  

IST1797I STRUCTURE TYPE = LIST                                           

IST1517I LIST HEADERS = 2048 - LOCK HEADERS = 0                          

IST1373I STORAGE ELEMENT SIZE = 256                                      

IST924I -------------------------------------------------------------    

IST1374I                            CURRENT     MAXIMUM  PERCENT         

IST1375I STRUCTURE SIZE              25600K      50176K    *NA*          

IST1376I STORAGE ELEMENTS                16       37356       0          

IST1377I LIST ENTRIES                     3        3751       0           

IST924I -------------------------------------------------------------    

IST1834I LIST DVIPA SYSNAME  TCPNAME    #ENTRIES    TGCOUNT SEQNUMBER    

IST1835I    1 10.91.1.1                                                  

IST1836I            SYS1     TCPCS             1          1              

IST1838I               LIST ENTRY KEYS:                                  

IST1839I                 02000000000000010023C33000000000                

IST1835I    2 10.91.1.1                                                   

IST1837I            SYS1     TCPCS             1                    9    

IST1838I               LIST ENTRY KEYS:                                  

IST1839I                 00000000000000000000000000000000                

IST314I END                                                              

 

In this example, the first list entry is for DVIPA address 10.91.1.1. The DVIPA is owned by System SYS1 and stack TCPCS. There is one entry for an SA. The TGCOUNT is the Takeover / Giveback count which represents the number of times that the DVIPA has moved. There is only one entry for an SA in this example but there could be multiples if there are multiple SAs associated with the DVIPA. Also, if IKEv1 is being used, there would be an entry for both the phase 1 and phase 2 SAs.

 

The second list entry is also for DVIPA address 10.91.1.1. This is a sequence number entry that is associated with a specific SA. The SEQNUMBER value indicates the current value of the shared sequence number.

 

The EZBDVIPA structure can be rebuilt with the command SETXCF START,REBUILD,STRNAME=EZBDVIPA . This command causes information to be gathered from all active SWSA enabled TCP/IP stacks within the sysplex. The information is used to rebuild the structure.

 

DVIPA takeover

 

The IKED running on behalf of the TCP/IP stack of the DVIPA owner is responsible for all SA negotiations for the DVIPA. The TCP/IP stack that owns the DVIPA is responsible for keeping the coupling facility updated with the information needed to re-establish an SA in the event of a DVIPA takeover, as well as SA sequence numbers.

 

The deactivation of a tunnel that is being distributed, must occur on the distributing stack. The distributor then ensures that delete messages are sent to all the targets which will cause the associated shadow tunnels to be deleted. The “ipsec -y deactivate” command can be used to deactivate a tunnel that is being distributed but the command must be issued for the stack that owns the DVIPA (not a target stack). An “ipsec -y deactivate” command done on a target system, specifying the tunnel ID (e.g. Yxx) of a shadow tunnel will typically fail. However, it could delete a non-shadow tunnel with the same tunnel ID.

 

When a DVIPA is moved during a DVIPA takeover (planned or unplanned), the new DVIPA owner attempts to re-establish SAs using the SA information stored in the CF for the DVIPA. The existing SAs for the DVIPA are deleted. If the new SAs are established successfully, IKED messages EZD1052I and EZD1097I are issued. Here is an example of these messages:

 

EZD1052I Phase 1 security association for DVIPA y.y.y.y is re-established with remote security endpoint z.z.z.z

EZD1097I Phase 2 security association for DVIPA y.y.y.y is re-established with remote security endpoint z.z.z.z

 

If an SA cannot be re-established, IKED messages EZD1036I and EZD1051I will be issued. Here is an example:

 

EZD1036I Phase 2 security association for DVIPA y.y.y.y is not re-established with remote security endpoint z.z.z.z

EZD1051I Phase 1 security association for DVIPA y.y.y.y is not re-established with remote security endpoint z.z.z.z

 

 

When an IKEv1 SA is re-established, the process is transparent to the remote peer that owns the other end of the SA. The process appears to be a normal SA refresh.

 

When an IKEv2 SA is re-established, the SA re-establishment appears as a new SA to the remote peer.  That is new IKESA and Child SA negotiations occur.

 

In some cases, the old SA on the peer system might not be deleted immediately. This can happen for SAs established for VIPARANGE DVIPAs. A new SA is not automatically re-established for a VIPARANGE DVIPA. The old SA can be deleted from the original owner without notification to the peer. When this scenario occurs, the old SA on the peer remains active until it is deleted by liveness checking, which is described in RFC 5996. For more information about liveness checking as implemented on the z/OS® platform, see the LivenessInterval parameter on the KeyExchangePolicy statement.

 

 

SWSA Best Practices

 

• Ensure that there are IPsec policies on all participating systems, including distributing stacks, target stacks and backup stacks.

• Certificates identifying hosts must be available on all distributing and backup stacks. This is most easily accomplished by sharing the SAF certificate repository between the processors in the sysplex.

• It is important that “common” certificates be used on all possible DVIPA owners that might re-establish an SA to a remote peer. Each possible DVIPA owner should reference the same local identity in their IPsec policy. When a DVIPA moves to a backup TCP/IP stack, the stack retrieves SA information for the DVIPA from the EZBDVIPA structure in the coupling facility (CF). The SA information includes the local identity which, along with other information, is used to locate a key exchange rule on the backup stack.

• If SWSA tunnels are deleted prior to dvipa movement to a backup, backups will not re-establish any tunnels.
   
• From a stack perspective, all anchor rules that are applicable to distributed DVIPA traffic must be identical on all systems. In addition, the ordering of the rules must allow for consistent application of security policy on all systems.
   
• To be considered a sysplex-wide SA, the negotiated SA must be at a granularity no coarser than host for the local DVIPA address. That is, the local endpoint for the SA cannot use a subnet or range that encompasses a DVIPA address. This ensures that on a DVIPA giveback that the SA can be moved from host to host without concerns about an SA being applicable to both the backup and primary host simultaneously. If an SA is negotiated with a subnet or range local endpoint that encompasses a DVIPA, the IPsec traffic using the SA cannot be distributed or recovered through the DVIPA takeover support.

 

 

Automation Best Practices

 

For a planned outage with DVIPAs moving to a backup, you should avoid a scenario that leads to orphaned SAs on the remote peer(s). The suggested shutdown order is:

 

   VARY TCPIP,,SYSLPLEX,LEAVEGROUP

• LEAVEGROUP will bring down all DVIPAs on the TCP/IP stack (including VIPARANGE DVIPAs) allowing the DVIPAs to move to a backup stack.
• Ensure that automation for moving applications that use VIPARANGE DVIPAs is done BEFORE the LEAVEGROUP.
• IPsec SAs for DVIPAs that move to a backup stack are re-established by the backup stack.

   IKED

   TRMD

   NSSD

   OMPROUTE       

   TCPIP

   PAGENT

   ICSF

   RESOLVER

   VTAM

 

Recommendation on Startup: Bring up ICSF, NSSD, and IKED before starting TCP/IP. When the TPCIP stack is brought up, it will regain ownership for any DVIPAs that it has defined. IKED, NSSD, and ICSF need to be ready to renegotiate tunnels for the DVIPA.

Note that Pagent has dependencies on ICSF for AT-TLS, if applicable

 

Here is the order for automation startup:
 

   RESOLVER

   VTAM

   ICSF

   PAGENT

   TCPIP

   OMPROUTE

   NSSD

   TRMD    

   IKED

 

 

Loss of access to coupling facility

 

If access is lost to the coupling facility that contains the DVIPA structure EZBDVIPA, it is possible the TCP connections that are using this DVIPA might stop and new connections that need IPsec will fail to establish. Loss of access can be caused by any of the following events:

 

•     A disconnect from the coupling facility structure.
•     The structure is being rebuilt.
•     The structure encounters a critical storage shortage.

 

Loss of coupling facility access should affect only sysplex distributor connections that are being encrypted or authenticated. When access to EZBDVIPA is restored, the sessions can be re-established.

 

 

Additional References

 

z/OS Communications Server: IP Configuration Guide, section “Sysplex-Wide Security Associations and IP security”

z/OS Communications Server: IP Configuration Reference, IPSEC statement, DVIPSEC and DVLOCALFLTR parameters.

z/OS Communications Server: IP Diagnosis Guide, section "Diagnosing sysplex-wide security association (SWSA) problems"