z/OS V2R3 introduces the ability to encrypt coupling facility (CF) structure data as part of the Pervasive Encryption solution umbrella with IBM Z. With this support, all customer information stored in CF structures can be encrypted while being transferred to and from the CF and while it resides at rest in the CF. CF List and Cache structures can contain customer data and can therefore be encrypted while Lock structures as well as Directory Only Cache structures do not and will not allow encryption.
There are several reasons why it may make sense to encrypt CF structure data, including the direction with data protection standards like the General Data Protection Regulation (GDPR) that encryption and other security measures that protect all personal data are expected to be utilized or face financial and reputational consequences when a data breach occurs. Furthermore, encryption and decryption of CF structure data is handled transparently by the z/OS host, meaning there are no application or middleware enablement support requirements to accomplish this CF structure data encryption. Additionally, since the encryption and decryption processing is done by IBM Z Server’s Protected key CPACF feature which provides high performance low latency encryption, there should be minimal impact to CF processing when the data is encrypted.
Coupling Facility exploitation in a Parallel Sysplex is a significant part of how we test z/OS and the Z system in our zPET environment and so we made CF structure encryption a central part of our test efforts for V2R3. We’d like to tell you about our approach.
Requirements & Coexistence
Requirements for enabling the CF structure data encryption support are documented in z/OS MVS Setting Up a Sysplex document for V2.3. You should refer there first to understand all of the requirements. The major requirements that affected our environment were:
From a software perspective:
- Before enabling CF structure data encryption, we ensured all systems in our sysplex were migrated to z/OS V2.3.
- Full support is provided when all systems are at V2.3
- Toleration support is provided on z/OS V2.1, V2.2 via APAR OA52060 for fall back scenarios, including supporting CFRM policies where CF structure data encryption has been activated.
- z/OS ICSF is required and is already a part of our environment. We tested with HCR77C0 & HCR77C1 levels of ICSF.
- No explicit application or middleware support is required, meaning that CF structure data encryption was transparent to our applications. We confirmed this for every CF structure type, and for most of the CF structure exploiters that we run in our environment. We’ll discuss this more later in this article.
From a hardware perspective,
- The CP Assist for Cryptographic Functions (CPACF) Protected Key Support feature is required and is already a part of our environment. We tested with CPACF enabled on zEC2, z13 and z14 processors in our environment.
- Crypto Express3 or later Coprocessor are required. IBM z14 and Crypto Express6 Coprocessors provides the best co-processor support so we did a good amount of our testing with the Crypto Express6 but we also tested with Crypto Express5 on IBM z14, Crypto Express5 on IBM z13 and Crypto Express3 and Crypto Express4 on IBM zEC12.
Enablement approach
Encryption is enabled on a structure by structure basis via the new ENCRYPT structure keyword in the CFRM policy definition. The Administrative Data Utility (IXCMIAPU) is used to create or update structure definitions in the CFRM policy data, and when run with a policy that contains structures that are defined with ENCRYPT(YES), the utility will create and assign secure cryptographic key tokens to a structure definition in the CFRM Couple Dataset (CDS). These secure cryptographic key tokens are obtained from ICSF. Rebuilding a structure with a pending policy change for the ENCRYPT state is all that is required to begin encrypting or decrypting the data in that structure. Encryption of the structure data is done using the CPACF feature with the secure cryptographic key tokens associated with the structure.
Here is an infographic that summarizes the steps to enable CF structure data encryption:
Anyone familiar with how to update CFRM policy structure definitions and make those changes active will recognize that encrypting a structure takes the same standard CFRM policy update procedure. The only difference is the new ENCRYPT parameter, provided the requirements are in place to support it.
There are further considerations to take into account when running with CFRM policies enabled for CF structure data encryption such as managing multiple administrative policies, for example for backups and DR, or for migrating between structure or CF modes.
You must ensure that all systems associated with the sysplex, including those used to format Couple Data sets, or GDPS K-systems, or contingency LPARs, or Disaster Recovery failovers systems, all use the same AES master key for encryption. You must also run the policy utility on a system with the same AES master key as the systems in the sysplex where the encrypted structures are to be used.
Consult the z/OS 2.3.0 MVS Setting Up the Sysplex publication on exactly what needs to be considered for your specific environment and CF configuration.
Since it’s required that all systems be at V2R3 before enabling CF structure encryption, we waited to enable it until we knew all our systems were at V2R3 permanently. We run a smaller test environment which allowed us to more easily migrate fully to V2R3, as well as test out CF structure encryption from an operational perspective, before focusing our efforts on our larger 15-way sysplex where more complex scenarios could be exercised such as data sharing applications exploiting encrypted data and heavy workload levels to stress the entire application stack while running those applications with encrypted data.
We also utilized the recently released version 1.8.1 of the zBNA capacity planning tool to estimate the cost of enabling CF structure data encryption on each of our systems. We recently added a blog entry about our experiences with this new zBNA support called zBNA for Pervasive Encryption Estimation.
Finally, we adopted a staged approach to encrypting structure data, focusing first on individual structures by type and by exploiters to test out the procedure and impact in our environment then broadening our focus to encrypt structures for an entire data sharing group and then finally focusing on encrypting all structures that support an application.
We’ll discuss the middleware structure encryption individually in separate blog posts, but in summary we encrypted structures for:
- IBM IMS V14
- IBM Db2 at V11 & V12
- IBM MQ
- IBM CICS
- In addition to z/OS infrastructure support structures such as the XCF signaling, Operlog and JES2 checkpoint structures.
Operating a sysplex with encrypted CF structure Data
We found that having CF structures with encrypted data to be transparent in normal operations in our parallel sysplexes. Structure policy changes, including changing the encryption state (with the new ENCRYPT(YES | NO | default=NO) ) policy keyword works seamlessly via the policy utility, then running the SETXCF START,POLICY command then rebuilding the affected structure.
We had no issues with managing and switching multiple policies that contain structures with encrypted data. We also saw no difference in switching in or out Couple Data Sets that contain those policies. We also were able to transparently change secure key tokens for structures with the commandSETXCF MODIFY,STRNM=strname,ENCRYPTKEY.
We found that we could monitor the state of encrypted structures in several ways including:
- DISPLAY XCF,STRUCTURE command
- RMF Coupling Facility Structure Details dialog in RMF Monitor III
- RMF CF Activity post processor report
Here’s a sample output from the D XCF,STR command for one of our XCF signaling structure which has been enabled for encryption. Note from the highlighted line that the POLICY information shows ENCRYPT : YES and the allocation information in the ACTIVE STRUCTURE reports an ENCRYPTION LEVEL: AES-256 PROTECTED KEY indicating that the policy definition requests encryption for this structure and that the structure has been rebuilt to enable encryption of the data sent to and stored in it.
D XCF,STR,STRNM=IXCPLEX_PATH6
IXC360I 15.09.09 DISPLAY XCF 978
STRNAME: IXCPLEX_PATH6
STATUS: ALLOCATED
EVENT MANAGEMENT: POLICY-BASED
TYPE: LIST
POLICY INFORMATION:
POLICY SIZE : 180224 K
POLICY INITSIZE: 95232 K
POLICY MINSIZE : 71424 K
FULLTHRESHOLD : 90
ALLOWAUTOALT : YES
REBUILD PERCENT: N/A
DUPLEX : DISABLED
ALLOWREALLOCATE: YES
PREFERENCE LIST: CF4 CF3 CF1 CF2
ENFORCEORDER : YES
EXCLUSION LIST IS EMPTY
ENCRYPT : YES
ACTIVE STRUCTURE
----------------
ALLOCATION TIME: 09/12/2017 17:02:41
CFNAME : CF4
COUPLING FACILITY: 002964.IBM.02.0000000840F7
PARTITION: 14 CPCID: 00
STORAGE CONFIGURATION ALLOCATED MAXIMUM %
ACTUAL SIZE: 94 M 176 M 53
SPACE USAGE IN-USE TOTAL %
ENTRIES: 2 20043 0
ELEMENTS: 33 20003 0
PHYSICAL VERSION: D32157DF 7FE67E92
LOGICAL VERSION: D32157DF 7FE67E92
SYSTEM-MANAGED PROCESS LEVEL: NOT APPLICABLE
DISPOSITION : DELETE
ACCESS TIME : 0
MAX CONNECTIONS: 32
# CONNECTIONS : 15
ENCRYPTION LEVEL: AES-256 PROTECTED KEY
CONNECTION NAME ID VERSION SYSNAME JOBNAME ASID STATE
---------------- -- -------- -------- -------- ---- ---------
SIGPATH_0A00529C 0A 000A015C JD0 XCFAS 0006 ACTIVE
SIGPATH_0C0052A3 0C 000C0160 JG0 XCFAS 0006 ACTIVE
SIGPATH_0D0052A0 0D 000D0138 JH0 XCFAS 0006 ACTIVE
SIGPATH_0100529B 01 000101E8 J90 XCFAS 0006 ACTIVE
SIGPATH_0200529E 02 000201EE JF0 XCFAS 0006 ACTIVE
SIGPATH_0300529F 03 00030247 JC0 XCFAS 0006 ACTIVE
SIGPATH_04005298 04 00040207 JB0 XCFAS 0006 ACTIVE
SIGPATH_050052A1 05 0005021D JA0 XCFAS 0006 ACTIVE
SIGPATH_060052A4 06 000601F9 Z0 XCFAS 0006 ACTIVE
SIGPATH_0700529A 07 00070215 JE0 XCFAS 0006 ACTIVE
SIGPATH_080052A2 08 000801FD TPN XCFAS 0006 ACTIVE
SIGPATH_090052A6 09 00090208 J80 XCFAS 0006 ACTIVE
SIGPATH_1100529D 11 00110125 JI0 XCFAS 0006 ACTIVE
SIGPATH_12005299 12 00120112 JJ0 XCFAS 0006 ACTIVE
SIGPATH_130052A5 13 00130138 JL0 XCFAS 0006 ACTIVE
DIAGNOSTIC INFORMATION: STRNUM: 000001B3 STRSEQ: 00000000
MANAGER SYSTEM ID: 0D0052A0
NAME/MGR #QUEUED 1STQESN LASTQESN CMPESN NOTIFYESN
J80 00000000 00000000 00000000 00000000 00000000
EVENT MANAGEMENT: MESSAGE-BASED MANAGER SYSTEM NAME: JH0
MANAGEMENT LEVEL : 01052010
If we displayed a structure that supports encryption but does not specify anything for ENCRYPT in the policy (or if you specified ENCRYPT : NO), you will still see that the ACTIVE STRUCTURE section reports ENCRYPTION LEVEL: NOT ENCRYPTED. Note that for this IXCPLEX1_PATH1 structure display, we’ve abbreviated the output for IXC360I by removing the connection details:
D XCF,STR,STRNM=IXCPLEX_PATH1
IXC360I 15.19.28 DISPLAY XCF 155
STRNAME: IXCPLEX_PATH1
STATUS: ALLOCATED
EVENT MANAGEMENT: POLICY-BASED
TYPE: LIST
POLICY INFORMATION:
POLICY SIZE : 2723840 K
POLICY INITSIZE: 1369088 K
POLICY MINSIZE : 1026816 K
FULLTHRESHOLD : 90
ALLOWAUTOALT : YES
REBUILD PERCENT: N/A
DUPLEX : DISABLED
ALLOWREALLOCATE: YES
PREFERENCE LIST: CF4 CF3 CF2 CF1
ENFORCEORDER : NO
EXCLUSION LIST IS EMPTY
ACTIVE STRUCTURE
----------------
ALLOCATION TIME: 08/07/2017 08:32:12
CFNAME : CF4
COUPLING FACILITY: 002964.IBM.02.0000000840F7
PARTITION: 14 CPCID: 00
STORAGE CONFIGURATION ALLOCATED MAXIMUM %
ACTUAL SIZE: 1339 M 2660 M 50
SPACE USAGE IN-USE TOTAL %
ENTRIES: 1 315507 0
ELEMENTS: 30 315494 0
PHYSICAL VERSION: D2F3A2A1 0C387BA7
LOGICAL VERSION: D2F3A2A1 0C387BA7
SYSTEM-MANAGED PROCESS LEVEL: NOT APPLICABLE
DISPOSITION : DELETE
ACCESS TIME : 0
MAX CONNECTIONS: 32
# CONNECTIONS : 15
ENCRYPTION LEVEL: NOT ENCRYPTED
Similarly, if we display the ISGLOCK structure which does not support encryption since it’s a Lock structure, you see that the ACTIVE STRUCTURE reports NOT APPLICABLE - NOT SUPPORTED FOR STRUCTURE TYPE. Note that for this ISGLOCK structure display, we’ve abbreviated the output for IXC360I to show only the ACTIVE STRUCTURE section:
IXC360I 14.10.17 DISPLAY XCF 115
STRNAME: ISGLOCK
STATUS: ALLOCATED
.
.
ACTIVE STRUCTURE
----------------
ALLOCATION TIME: 08/01/2017 01:50:58
CFNAME : CF4
COUPLING FACILITY: 002964.IBM.02.0000000840F7
PARTITION: 14 CPCID: 00
STORAGE CONFIGURATION ALLOCATED MAXIMUM %
ACTUAL SIZE: 65 M 65 M 100
SPACE USAGE TOTAL
LOCKS: 8388608
PHYSICAL VERSION: D2EBBDC1 3A915CAD
LOGICAL VERSION: D2EBBDC1 3A915CAD
SYSTEM-MANAGED PROCESS LEVEL: 8
XCF GRPNAME : IXCLO0B0
DISPOSITION : DELETE
ACCESS TIME : 0
MAX CONNECTIONS: 32
# CONNECTIONS : 15
ENCRYPTION LEVEL: NOT APPLICABLE - NOT SUPPORTED FOR STRUCTURE TYPE
RMF Monitor III Coupling Facility Activity dialog now includes a new column E which will show whether the structure was encrypted (with a value Y), not encrypted (with a value N) or not applicable for encryption (with value of - ) in the monitored interval being displayed.
RMF V2R3 CF Activity - UTCPLXJ8
Samples: 60 Systems: 15 Date: 12/02/17 Time: 15.10.00 Range: 60 Sec
CF: ALL Type ST E System CF --- Sync --- -------- Async -------
Util Rate Avg Rate Avg Chg Del
Structure Name % Serv Serv % %
.
.
ISGLOCK LOCK A - *ALL 0.6 2932 11 0.0 0 0.0 0.0
.
.
IXCPLEX_PATH1 LIST A N *ALL 1.2 0.0 0 267.9 118 0.0 0.0
IXCPLEX_PATH2 LIST A N *ALL 17.1 0.0 0 29207 33 0.0 0.0
IXCPLEX_PATH3 LIST A N *ALL 39.6 0.0 0 11005 44 0.0 0.0
IXCPLEX_PATH4 LIST A Y *ALL 2.5 0.0 0 3581 39 0.0 0.0
IXCPLEX_PATH5 LIST A Y *ALL 42.9 0.0 0 15975 36 0.0 0.0
IXCPLEX_PATH6 LIST A Y *ALL 45.8 0.0 0 16356 38 0.0 0.0
.
.
The RMF CF Activity post processor report has also been updated to indicate Encryption state for structures. In the GENERAL STRUCTURE SUMMARY section for the COUPLING FACILITY USAGE SUMMARY report section, a new ENC column is added to show whether the structure was encrypted, not encrypted or N/A(not applicable) for encryption, in the interval of time that this CF Activity report was created for.
Similarly, the COUPLING FACILITY STRUCTURE ACTIVITY section will also now display the ENCRYPTED = state, such as the following screen caption:
To date, once we completed the steps to enabled the CF structure data encryption and could confirm the encrypted states of our structures, the usage of structures with encrypted data has not presented any challenges for us. We have done testing with recovery of structure connectors, ICSF, z/OS and CFs as well.
We will evaluate changing the AES master key and the protected data keys as part of a larger key management strategy on some regular basis but our experiences so far have shown that those activities can be accomplished seamlessly with the provided commands and procedures from z/OS XCF and ICSF.
If you have any feedback or questions related to what we have done or what you would like to hear more of with regards to our experiences with CF Structure data encryption, please let us know via email.
Author: Kieron Hinds (kdhinds@us.ibm.com)