Configuring secure communications between z/OS components and Z APM Connect Distributed Gateway in OpenShift or Kubernetes clusters

If you installed the Z APM Connect DG in the Red Hat OpenShift clusters or Kubernetes clusters, configure z/OS® components machine in order for TLS connections to the Z APM Connect DG to work.

A complete configuration example is provided to enable secure communications between the OpenShift Z APM Connect DG machine and MVS host, using RACF as the Security Server. For more information, see (OpenShift) Security enablement example.

Before you begin

Make sure that the Secrets for Kafka, the Connection Manager, and Z APM Connect DG are created, and TLS is enabled with both the Connection Manager and Kafka routes in order for the OpenShift router or the Kubernetes router to properly direct traffic. Without TLS, there is no SNI header indicating the desired host name. For more information, see Pre-deployment configuration.

Procedure

  1. Transfer certificates from Z APM Connect DG OpenShift or Kubernetes clusters to z/OS components machine, create a key ring and certificate on z/OS components, and export personal certificate of the z/OS components.
    1. Use the ASCII transfer mode to FTP the CA certificate used to sign the Z APM Connect DG certificates from the Z APM Connect DG OpenShift or Kubernetes clusters to your z/OS machines.
      Important: You should be able to browse the certificate, and it should look the same as if you were to “cat” the CA certificate in the Z APM Connect DG OpenShift clusters or Kubernetes clusters.
    2. Create a key ring and certificate on z/OS components, and export personal certificate of the z/OS components.
    In the following command example:
    • <dsn_name> is the data set name that contains the root CA certificate.
    • WITHLABEL('<label>') is optional. The label should be unique, if used.
    • Commands use a <keyring name> owned by IBMUSER. For more information about creating a key ring, see RACDCERT ADDRING.
    • The UserId under which the z/OS Connect EE server and Z APM Connect Base run is IBMUSER.
    • The UserId under which the Z APM Connect Base address space (AGMAPROC by default) runs is USER1.
    • - is the continuation character.
    Example
    RACDCERT CERTAUTH ADD('<dsn_name>') HIGHTRUST WITHLABEL('<label>')
    
    SETROPTS RACLIST(DIGTCERT) REFRESH 
    
    RACDCERT CERTAUTH GENCERT                              -
       SUBJECTSDN(O('<organization name>')                 -
                  OU('<organizational unit name>')         -
                   C('<country>'))                         -
                   KEYUSAGE(CERTSIGN)                      -
                   WITHLABEL('<Internal CA Name>'))
    
    SETROPTS RACLIST(DIGTCERT) REFRESH
    
    RACDCERT ID(IBMUSER) GENCERT SUBJECTSDN(CN('<common name>')                                                   -
    								      O('<organization name>')                -
    									  OU('<organizational unit name>')    -
    									  C('<country>'))                     -
    					WITHLABEL('<IBMUSER Personal Certificate name>')                      -
    					SIGNWITH(CERTAUTH LABEL('<Internal CA Name>'))
    
    SETROPTS RACLIST(DIGTCERT) REFRESH
    
    RACDCERT ID(IBMUSER) ADDRING(<keyring name>)
    
    RACDCERT ID(IBMUSER) CONNECT(CERTAUTH                                   -
                               LABEL('<Internal CA Name>')                  -
                               RING(<keyring name>)                         -
                               USAGE(CERTAUTH))
    
    RACDCERT ID(IBMUSER) CONNECT(ID(IBMUSER)                                -
                               LABEL('<IBMUSER Personal Certificate name>') -
                               RING(<keyring name>)                         -
                               DEFAULT                                      -
                               USAGE(PERSONAL))
    
    RACDCERT ID(IBMUSER) CONNECT(CERTAUTH                                   -
                               LABEL('<lable>')                  -
                               RING(<keyring name>)                         -
                               USAGE(CERTAUTH))
    
    
    SETROPTS RACLIST(DIGTRING) REFRESH
    SETROPTS RACLIST(DIGTCERT) REFRESH
    
    
    PERMIT IRR.DIGTCERT.LISTRING CLASS(FACILITY) ID(IBMUSER) ACCESS(READ)
    PERMIT IRR.DIGTCERT.GENCERT CLASS(FACILITY) ID(IBMUSER) ACCESS(CONTROL)
    
    SETROPTS RACLIST(FACILITY) REFRESH
      
    RACDCERT CERTAUTH EXPORT(LABEL('<unique name>')) DSN('<dsn_export>')
    To give other users access to the key ring, (for instance, if the Z APM Connect Base address space is running under a UserId other than IBMUSER, in this example USER1), issue the following three commands, then also specify the LABEL value in the pagent configuration for that proc:
    PERMIT IRR.DIGTCERT.LISTRING CLASS(FACILITY) ID(USER1) ACCESS(UPDATE)
     
    RACDCERT ID(USER1) GENCERT SUBJECTSDN(CN('<common name>')                                                   -
    								      O('<organization name>')                -
    									  OU('<organizational unit name>')    -
    									  C('<country>'))                     -
    					WITHLABEL('<IBMUSER Personal Certificate name>')                      -
    					SIGNWITH(CERTAUTH LABEL('<Internal CA Name>'))
    
    RACDCERT ID(IBMUSER) CONNECT(ID(USER1)                                -
                               LABEL('<IBMUSER Personal Certificate name>') -
                               RING(<keyring name>)                         -     
                               USAGE(PERSONAL))
    
  2. Update the policy agent (PAGENT) AT-TLS policy on the MVS system.
    Depending on the type of transactions to be tracked, there are up to 3 components on z/OS that need to connect to the Z APM Connect Distributed Gateway:
    1. The Z APM Connect Base address space (named AGMAPROC by default) needs to connect to the ingress.apps process of OpenShift or Kubernetes. You need to set a TTLSRule to define the rules for this connection. See the TTLSRule OpenShift-ingress rule below for an example.
    2. If you are tracking transactions through z/OS Connect EE to CICS or IMS, the z/OS Connect EE server process needs to connect to the kafka.apps process of OpenShift or Kubernetes. You need to set a separate TTLSRule to define the rules for this connection. See the TTLSRule OpenShift-kafka rule below for an example.
    3. Similar to b, if you are tracking transactions through the CICS Transaction Gateway (CTG) to CICS, the CTG server process needs to connect to the kafka.apps process of OpenShift or Kubernetes. You need to set a separate TTLSRule to define the rules for this connection. See the TTLSRule OpenShift-kafka rule below for an example.
      Tip: If you have both z/OS Connect and CTG, the two servers can share the same TTLSRule. You need to either remove the optional Jobname statement, or use a wildcard to set the Jobname so that it works for both the z/OS Connect and CTG processes.
    Note:
    • The ingress.apps and kafka.apps prefix values are configurable on the DG, so may be something other than ingress.apps and kafka.apps.
    • If your installation uses Integrated Cryptographic Service Facility (ICSF), you need to authorize the z/OS components (Z APM Connect Base proc, z/OS Connect EE and CTG servers) to use ICSF.
    • To troubleshoot AT-TLS connections between z/OS and the DG, enable Trace 255 in the policy agent to surface errors on the system log.
    In the following command example:
    • The z/OS Connect EE server proc is ZCEESRVR.
    • The Z APM Connect Base proc is AGMAPROC.
    • The OpenShift machine name is ZAPMC-OSDG.ibm.com.
    • If you have a CICS TG server, configure its connection to the Z APM Connect DG machine the same as you do for the z/OS Connect EE server.

    Example

    TTLSRule                      OpenShift-kafka
    {
        RemotePortRange 443
        RemoteAddr x.xx.xx.xxx # <-- ip address of OpenShift machine
        Jobname ZCEEPROC # <-- zOS Connect EE server proc name, or CICS TG server proc name
        Direction Outbound
        TTLSGroupActionRef       grp_Production
        TTLSEnvironmentActionRef act_OpenShift
        TTLSConnectionActionRef   act_OpenShift-kafka_Conn
    }
    
    TTLSConnectionAction act_OpenShift-kafka_Conn
    {
        HandshakeRole     Client
       TTLSConnectionAdvancedParms
       {
           ClientHandshakeSNI Optional
           ClientHandshakeSNIMatch Optional
           ClientHandshakeSNIList kafka.apps.xyz.fyre.ibm.com # <-- kafka.apps.<fully.qualified.OpenShift.machine.name>
           SSLv3 Off
        }
    }
    
    TTLSEnvironmentAction         act_OpenShift
    {
        HandshakeRole Client
        Trace   255
        TTLSKeyRingParms
        {
            Keyring IBMUSER/OPENSHIFT-RING	# <-- Keyring owner/Keyring Name
         }
        TTLSEnvironmentAdvancedParms
        {
            SecondaryMap On
            HandshakeTimeout 60
            TLSV1.2 On            
        }
    }
    
    
    TTLSRule                      OpenShift-ingress
    {
        RemotePortRange 443
        RemoteAddr x.xx.xx.xxx # <-- ip address of OpenShift machine
        Jobname AGMAPROC # <-- Z APM Connect Base proc name
        Direction Outbound
        TTLSGroupActionRef       grp_Production
        TTLSEnvironmentActionRef act_OpenShift
        TTLSConnectionActionRef   act_OpenShift-ingress_Conn
    }
    
    TTLSConnectionAction act_OpenShift-ingress_Conn
    {
       HandshakeRole     Client
       TTLSConnectionAdvancedParms
       {
           ClientHandshakeSNI Optional
           ClientHandshakeSNIMatch Optional
           ClientHandshakeSNIList ingress.apps.xyz.fyre.ibm.com   # <-- ingress.apps.<fully.qualified.OpenShift.machine.name>
           SSLv3 Off
        }
    }
    
  3. Transfer over FTP using the ASCII mode the exported certificate <dsn_export> to the existing truststore on the Z APM Connect DG OpenShift or Kubernetes clusters and place it in the same directory as the truststore file. The .cert file must be added to the truststore.
    keytool -import -keystore k8s.truststore.jks -deststoretype jks -alias customcert -file <dsn_export> -storepass <pw> -keypass <pw> -noprompt
  4. Recycle the Z APM Connect DG, if needed, to pick up the truststore addition.
    1. Run the following command to uninstall the Z APM Connect DG:
      helm uninstall <name>
      For example.
      helm uninstall zapm
    2. Run the following command to reinstall Z APM Connect DG.
      helm install --namespace <desired project> -f <values yaml> <name> <chart location>
      For example,
      helm install --namespace ibm-zapm -f values.yaml zapm ./zapm-connect-6.1.0.7.tgz
  5. Refresh or recycle the PAGENT address space. Recycle Z APM Connect Base address space (AGMAPROC) and if applicable, the CICS TG and z/OS Connect EE servers to pick up the new policy.