Configuring secure communications between z/OS components and Z APM Connect Distributed Gateway in OpenShift or Kubernetes clusters

If you installed the Z APM Connect DG in the Red Hat OpenShift clusters or Kubernetes clusters, configure z/OS® components machine in order for TLS connections to the Z APM Connect DG to work.

A complete configuration example is provided to enable secure communications between the OpenShift Z APM Connect DG machine and MVS host, using RACF as the Security Server. For more information, see (OpenShift) Security enablement example.

Before you begin

Make sure that the Secrets for Kafka, the Connection Manager, and Z APM Connect DG are created, and TLS is enabled with both the Connection Manager and Kafka routes in order for the OpenShift router or the Kubernetes router to properly direct traffic. Without TLS, there is no SNI header indicating the desired host name. For more information, see Pre-deployment configuration.

Procedure

Transfer certificates from Z APM Connect DG OpenShift or Kubernetes clusters to z/OS components machine, create a key ring and certificate on z/OS components, and export personal certificate of the z/OS components.

Use the ASCII transfer mode to FTP the CA certificate used to sign the Z APM Connect DG certificates from the Z APM Connect DG OpenShift or Kubernetes clusters to your z/OS machines.

Important: You should be able to browse the certificate, and it should look the same as if you were to “cat” the CA certificate in the Z APM Connect DG OpenShift clusters or Kubernetes clusters.
Create a key ring and certificate on z/OS components, and export personal certificate of the z/OS components.

In the following command example:

<dsn_name> is the data set name that contains the root CA certificate.
WITHLABEL('<label>') is optional. The label should be unique, if used.
Commands use a <keyring name> owned by IBMUSER. For more information about creating a key ring, see RACDCERT ADDRING.
The UserId under which the z/OS Connect EE server and Z APM Connect Base run is IBMUSER.
The UserId under which the Z APM Connect Base address space (AGMAPROC by default) runs is USER1.
- is the continuation character.

Example

RACDCERT CERTAUTH ADD('<dsn_name>') HIGHTRUST WITHLABEL('<label>')

SETROPTS RACLIST(DIGTCERT) REFRESH 

RACDCERT CERTAUTH GENCERT                              -
   SUBJECTSDN(O('<organization name>')                 -
              OU('<organizational unit name>')         -
               C('<country>'))                         -
               KEYUSAGE(CERTSIGN)                      -
               WITHLABEL('<Internal CA Name>'))

SETROPTS RACLIST(DIGTCERT) REFRESH

RACDCERT ID(IBMUSER) GENCERT SUBJECTSDN(CN('<common name>')                                                   -
								      O('<organization name>')                -
									  OU('<organizational unit name>')    -
									  C('<country>'))                     -
					WITHLABEL('<IBMUSER Personal Certificate name>')                      -
					SIGNWITH(CERTAUTH LABEL('<Internal CA Name>'))

SETROPTS RACLIST(DIGTCERT) REFRESH

RACDCERT ID(IBMUSER) ADDRING(<keyring name>)

RACDCERT ID(IBMUSER) CONNECT(CERTAUTH                                   -
                           LABEL('<Internal CA Name>')                  -
                           RING(<keyring name>)                         -
                           USAGE(CERTAUTH))

RACDCERT ID(IBMUSER) CONNECT(ID(IBMUSER)                                -
                           LABEL('<IBMUSER Personal Certificate name>') -
                           RING(<keyring name>)                         -
                           DEFAULT                                      -
                           USAGE(PERSONAL))

RACDCERT ID(IBMUSER) CONNECT(CERTAUTH                                   -
                           LABEL('<lable>')                  -
                           RING(<keyring name>)                         -
                           USAGE(CERTAUTH))


SETROPTS RACLIST(DIGTRING) REFRESH
SETROPTS RACLIST(DIGTCERT) REFRESH


PERMIT IRR.DIGTCERT.LISTRING CLASS(FACILITY) ID(IBMUSER) ACCESS(READ)
PERMIT IRR.DIGTCERT.GENCERT CLASS(FACILITY) ID(IBMUSER) ACCESS(CONTROL)

SETROPTS RACLIST(FACILITY) REFRESH
  
RACDCERT CERTAUTH EXPORT(LABEL('<unique name>')) DSN('<dsn_export>')

To give other users access to the key ring, (for instance, if the Z APM Connect Base address space is running under a UserId other than IBMUSER, in this example USER1), issue the following three commands, then also specify the LABEL value in the pagent configuration for that proc:

PERMIT IRR.DIGTCERT.LISTRING CLASS(FACILITY) ID(USER1) ACCESS(UPDATE)
 
RACDCERT ID(USER1) GENCERT SUBJECTSDN(CN('<common name>')                                                   -
								      O('<organization name>')                -
									  OU('<organizational unit name>')    -
									  C('<country>'))                     -
					WITHLABEL('<IBMUSER Personal Certificate name>')                      -
					SIGNWITH(CERTAUTH LABEL('<Internal CA Name>'))

RACDCERT ID(IBMUSER) CONNECT(ID(USER1)                                -
                           LABEL('<IBMUSER Personal Certificate name>') -
                           RING(<keyring name>)                         -     
                           USAGE(PERSONAL))

Update the policy agent (PAGENT) AT-TLS policy on the MVS system.

In the following command example:

The z/OS Connect EE server proc is ZCEESRVR.
The Z APM Connect Base proc is AGMAPROC.
The OpenShift machine name is ZAPMC-OSDG.ibm.com.
If you also have a CICS TG server, configure its connection to the Z APM Connect DG machine the same as you do for the z/OS Connect EE server.

Note: The Z APM Connect Base address space connects to the ingress.apps process of OpenShift or Kubernetes, and the z/OS Connect EE and CICS TG servers (if applicable) connect to the kafka.apps process of OpenShift or Kubernetes. These prefix values are configurable, so may be something other than ingress.apps and kafka.apps.

Example

TTLSRule                      OpenShift-kafka
{
    RemotePortRange 443
    RemoteAddr x.xx.xx.xxx # <-- ip address of OpenShift machine
    Jobname ZCEEPROC # <-- zOS Connect EE server proc name, or CICS TG server proc name
    Direction Outbound
    TTLSGroupActionRef       grp_Production
    TTLSEnvironmentActionRef act_OpenShift
    TTLSConnectionActionRef   act_OpenShift-kafka_Conn
}

TTLSConnectionAction act_OpenShift-kafka_Conn
{
    HandshakeRole     Client
   TTLSConnectionAdvancedParms
   {
       ClientHandshakeSNI Optional
       ClientHandshakeSNIMatch Optional
       ClientHandshakeSNIList kafka.apps.xyz.fyre.ibm.com # <-- kafka.apps.<fully.qualified.OpenShift.machine.name>
       SSLv3 Off
    }
}

TTLSEnvironmentAction         act_OpenShift
{
    HandshakeRole Client
    Trace   255
    TTLSKeyRingParms
    {
        Keyring IBMUSER/OPENSHIFT-RING	# <-- Keyring owner/Keyring Name
     }
    TTLSEnvironmentAdvancedParms
    {
        SecondaryMap On
        HandshakeTimeout 60
        TLSV1.2 On            
    }
}


TTLSRule                      OpenShift-ingress
{
    RemotePortRange 443
    RemoteAddr x.xx.xx.xxx # <-- ip address of OpenShift machine
    Jobname AGMAPROC # <-- Z APM Connect Base proc name
    Direction Outbound
    TTLSGroupActionRef       grp_Production
    TTLSEnvironmentActionRef act_OpenShift
    TTLSConnectionActionRef   act_OpenShift-ingress_Conn
}

TTLSConnectionAction act_OpenShift-ingress_Conn
{
   HandshakeRole     Client
   TTLSConnectionAdvancedParms
   {
       ClientHandshakeSNI Optional
       ClientHandshakeSNIMatch Optional
       ClientHandshakeSNIList ingress.apps.xyz.fyre.ibm.com   # <-- ingress.apps.<fully.qualified.OpenShift.machine.name>
       SSLv3 Off
    }
}

Transfer over FTP using the ASCII mode the exported certificate <dsn_export> to the existing truststore on the Z APM Connect DG OpenShift or Kubernetes clusters and place it in the same directory as the truststore file. The .cert file must be added to the truststore.
```
keytool -import -keystore k8s.truststore.jks -deststoretype jks -alias customcert -file <dsn_export> -storepass <pw> -keypass <pw> -noprompt
```
Recycle the Z APM Connect DG, if needed, to pick up the truststore addition.
1. Run the following command to uninstall the Z APM Connect DG:
  helm uninstall <name>
  For example.
```
helm uninstall zapm
```
2. Run the following command to reinstall Z APM Connect DG.
```
helm install --namespace <desired project> -f <values yaml> <name> <chart location>
```
  For example,
```
helm install --namespace ibm-zapm -f values.yaml zapm ./zapm-connect-6.1.0.7.tgz
```
Refresh or recycle the PAGENT address space. Recycle Z APM Connect Base address space (AGMAPROC) and if applicable, the CICS TG and z/OS Connect EE servers to pick up the new policy.