Deploying a GREP11 server as a Hyper Protect Container Runtime instance

Complete the following steps to integrate with GREP11 server.

  1. Prepare GREP11 server image
  2. Prepare configuration files for c16 client
  3. Prepare configuration files for GREP11 server
  4. Deploy GREP11 server
  5. Configure GREP11 client with enabled mTLS authentication

1. Prepare GREP11 server image

Load the GREP11 server image from the tar archive. If you don't have the tar archive, see Downloading Crypto Express Network API for Secure Execution Enclaves.

Record the tag of the image, which is required to construct docker-compose.yml in Step 4.1.

docker load -i grep11server_s390x_1.1.2.1.tar

The following snippet shows an example of the output:

Loaded image: sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.1.2

2. Prepare configuration files for c16 client

To prepare and collect the following files that are required for connecting the c16 client to the c16 server, see Generate server certificate (openssl) with the generated server CSR to be assigned by CA ROOT. Use these values to prepare the configuration file.

Filename Description
c16client.yaml c16 client configuration file
c16server-ca.pem c16server CA root certificate (used as CA root on client side in mTLS connection with c16server)
c16server-client.key c16server client private key (used as client key on client side in mTLS connection with c16server)
c16server-client.pem c16server client certificates (used as client certificate on client side in mTLS connection with c16server)
c16server-restricted-server.pem c16server server certificate (used to do restricted server certificate validation, which will ensure only the permitted target server (which is the same as the one used on the c16server side) can be accessed)

Sample of c16client.yaml as below, and variables are required to replace:

#trace, debug, info, warn, err, error, critical, off
loglevel: 'debug'
servers:
  - hostname: ${C16SERVER_IP}
    port: 9001
    mTLS: true
    server_cert_file: "/cfg/c16server-ca.pem"
    client_key_file: "/cfg/c16server-client.key"
    client_cert_file: "/cfg/c16server-client.pem"

Where C16SERVER_IP: Is the IP address of the c16 server.

Note:

  1. If no restrict_server_cert_file parameter is set or it is empty, then c16client will work normally.
  2. If you want to restrict the server accessibility, set the parameter restrict_server_cert_file to the right c16server certificate.

Trouble Shooting: If restrict_server_cert_file is set, and you see the following error message from the server:

grep11server-ep11server-1  | [c16client][error] c16client::c16OpenAdapter: Failed to verify the identity of target server, please re-config and connect with permitted server.

Ensure that you do the following:

  1. Set the parameter restrict_server_cert_file with the correct c16server certificate.
  2. Connect to the target c16server.

3. Prepare configuration files for GREP11 server

To prepare and collect the following certificates that are required for configuring the GREP11 server, see Creating OpenSSL certificates for GREP11 Servers.

Filename Description
grep11server.yaml GREP11 server configuration file
grep11-ca.pem GREP11 server CA certificates(used as CA root on GREP11server side in mTLS connection)
grep11-server.key GREP11 server private key(used as server key on GREP11server side in mTLS connection)
grep11-server.pem GREP11 server certificates(used as server certificate on GREP11server side in mTLS connection)

HSM domain values of c16server can be acquired from this step Configuring Crypto Express Card Domain.

Sample of grep11server.yaml:

#
# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#
logging:
  # Package log levels
  # Levels: info, warning, debug, error, fatal, trace, and panic
  levels:
    # example below
    # entry: debug
###################################
## GRPC SERVICES TALKING TO GREP11 server ##
ep11crypto:
  enabled: true
  connection:
    address: 0.0.0.0
    port: 9876
    # Secure connection TLS options
    tls:
      enabled: true
      # certfile, keyfilie and cacert refer to the pem files that holds the certs
      certfile: /cfg/grep11-server.pem
      keyfile: /cfg/grep11-server.key
      mutual: true
      cacert: /cfg/grep11-ca.pem
      # same as above, but instead of filename, contents of PEM can be in an environment
      # variable (i.e. to avoid mounting files into docker container)
      cacertbytes:
      certfilebytes:
      keyfilebytes:
    # Server TCP/IP connection monitoring
    # serverKeepaliveTime is the duration in seconds after which if the server
    # does not see any activity from the client it pings the client to see
    # if it is alive
    # serverKeepaliveTimeout is the duration the server waits for a response
    # from the client after sending a ping before closing the connection
    keepalive:
      serverKeepaliveTime: 30
      serverKeepaliveTimeout: 5
  # Comma-separated list of card.domain tuples. card and domain are hex numbers
  # i.e. "01.a9,c.10" correspods to 169th domain on first card and 16th domain on 12th card
  domain: "${C16SERVER_HSM_DOMAIN}"

Where C16SERVER_HSM_DOMAIN: HSM domains, like "00.003a,01.003a", are configured on the c16server, maximum 2 domains are supported.

4. Deploy GREP11 server

GREP11 server can be deployed either on a local s390x environment or as a Hyper Protect Container Runtime instance.

4.1 Prepare docker-compose.yml

Prepare docker-compose.yml, which is used to boot up the GREP11 server container.

Note: You must change the image tag to grep11server image

Sample of docker-compose.yml:

services:
  ep11server:
    user: "0"
    image: ${GREP11SERVER_IMAGE_TAG}
    ports:
      - 9876:9876
    volumes:
      - ./c16client.yaml:/etc/c16/c16client.yaml
      - ./c16-ca.pem:/cfg/c16server-ca.pem
      - ./c16-client.key:/cfg/c16server-client.key
      - ./c16-client.pem:/cfg/c16server-client.pem
      - ./grep11server.yaml:/etc/ep11server/ep11server.yaml
      - ./grep11-ca.pem:/cfg/grep11-ca.pem
      - ./grep11-server.pem:/cfg/grep11-server.pem
      - ./grep11-server.key:/cfg/grep11-server.key

Where GREP11SERVER_IMAGE_TAG is the grep11server image tag, like "us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1@sha256:54334c919a93c45f70d0252d8518d24b64255a88a073b35239f95d056d444ab8"

For single or multiple containers, you can also use the play subsection. In the play subsection, you can define the workload via Pod descriptors. Each pod can contain one or more container definitions. Descriptors can be provided as shown in the following example:

apiVersion: v1
kind: Pod
metadata:
  name: grep11server
spec:
  securityContext:
    privileged: true
  containers:
  - name: grep11server
    image: us.icr.io/crypto-appliance-test/sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.0.0@sha256:7a6e20bd9c7aa5bc155212247859fcd5ee01d0005d123561e68fec3fa4452abe
    volumeMounts:
    - name: c16-client
      mountPath: /etc/c16/c16client.yaml
    - name: c16-ca
      mountPath: /cfg/c16server-ca.pem
    - name: c16-client-key
      mountPath: /cfg/c16server-client.key
    - name: c16-client-pem
      mountPath: /cfg/c16server-client.pem
    - name: grep11-config
      mountPath: /etc/ep11server/ep11server.yaml
    - name: grep11-ca
      mountPath: /cfg/grep11-ca.pem
    - name: grep11-server-pem
      mountPath: /cfg/grep11-server.pem
    - name: grep11-server-key
      mountPath: /cfg/grep11-server.key
    securityContext:
      privileged: true
    ports:
    - containerPort: 9876
      hostPort: 9876
  volumes:
  - name: c16-client
    hostPath:
      path: ./c16client.yaml
      type: FileOrCreate
  - name: c16-ca
    hostPath:
      path: ./c16-ca.pem
      type: FileOrCreate
  - name: c16-client-key
    hostPath:
      path: ./c16-client.key
      type: FileOrCreate
  - name: c16-client-pem
    hostPath:
      path: ./c16-client.pem
      type: FileOrCreate
  - name: grep11-config
    hostPath:
      path: ./grep11server.yaml
      type: FileOrCreate
  - name: grep11-ca
    hostPath:
      path: ./grep11-ca.pem
      type: FileOrCreate
  - name: grep11-server-pem
    hostPath:
      path: ./grep11-server.pem
      type: FileOrCreate
  - name: grep11-server-key
    hostPath:
      path: ./grep11-server.key
      type: FileOrCreate
  restartPolicy: Always

Run the following command to generate the base64 value of compose.tgz:

tar -czvf compose.tgz docker-compose.yml c16client.yaml grep11server.yaml c16-ca.pem c16-client.pem c16-client.key grep11-ca.pem grep11-server.key grep11-server.pem
base64 -i compose.tgz | tr -d '\n' > compose.b64

Prepare the contract file as shown in the following example:

env: |
  type: env
  logging:
    logDNA:
      hostname: syslog-a.us-south.logging.cloud.ibm.com
      ingestionKey: ab00e3....
      port: 6514
  volumes:
    test0:
      seed: stsolutiontest
workload: |
  type: workload
  play:
    archive:  ${COMPOSE_VALUE}
  auths:
    us.icr.io:
      username: iamapikey
      password: Eqx0TS....
  volumes:
    test0:
      mount: /mnt/data0
      seed: stsolutiontest
      filesystem: ext4

4.2 Deploy GREP11 server container locally

You can deploy the GREP11 server container on a local s390x server by running the following command.

docker-compose up

4.3 Deploy GREP11 server as a Hyper Protect Container Runtime instance

A Secure Execution VM can be deployed on premises or on Cloud. Both of them use the same steps to prepare contract. For more information, see Hyper Protect Container Runtime.

Complete the following steps to deploy the GREP11 server:

  • 4.3.1. Generate compose package
  • 4.3.2. Create a logDNA instance
  • 4.3.3. Prepare contract
  • 4.3.4. Create Secure Execution VM
  • 4.3.5. Verify GREP11 server deployment
  • 4.3.6. View log from logDNA instance console

4.3.1 Generate compose package

Complete the following steps to generate the compose package.

  1. Upload grep11server image to a docker registry

    To build Secure Execution VM, it is required to upload the grep11server image to a docker registry. Both us.icr.io and https://index.docker.io/v1/ are supported docker registries. For example,

    API_KEY=<API-KEY>
    docker login --username iamapikey -p $API_KEY us.icr.io
    docker tag sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.1.2.1 us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1
    docker push us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1
    
    USERNAME=<USERNAME>
    PASSWORD=<PASSWORD>
    docker login --username $USERNAME -p $PASSWORD docker.io
    docker tag sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.1.2.1 docker.io/hexiaohu/grep11server:s390x-1.1.2.1
    docker push docker.io/hexiaohu/grep11server:s390x-1.1.2.1
    
  2. Update the image tag in docker-compose.yml

    Update the image tag in docker-compose.yml to the value that reflects the image tag of docker registry. For example,

    ...
        image: us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1@sha256:54334c919a93c45f70d0252d8518d24b64255a88a073b35239f95d056d444ab8
    ...
    
  3. Generate base64 value of compose.tgz

    Compact all of config files together with docker-compose.yml into compose.tgz and get the base64 value of compose.tgz. Note: Ensure that the compose.tgz file contains only directories and regular files. Links or pipes are not supported.

    rm -f compose.tgz compose.b64
    tar -czvf compose.tgz docker-compose.yml c16client.yaml grep11server.yaml c16server-ca.pem c16server-client.pem c16server-client.key grep11-ca.pem grep11-server.key grep11-server.pem
    base64 -i compose.tgz | tr -d '\n' > compose.b64
    

4.3.2 Create a logDNA instance

Create a logDNA instance on IBM Cloud and get the following access point value Ingestion Key. For more information, see Getting started with Log Analysis and Getting ingestion keys.

Ingestion Key: <Ingestion Key Value>
Service Endponit: <Endpoint of service instance, eg. syslog-a.us-south.logging.cloud.ibm.com:6514>

4.3.3 Prepare contract

  • Prepare workload.yaml, the following is an example:

    Update the auths section to indicate the logon method of the docker registry, in which the grep11server image is located. Using us.icr.io as an example:

    type: workload
    auths:
      us.icr.io:
        password: ${API_KEY}
        username: iamapikey
    compose:
      archive: ${COMPOSE_VALUE}
    volumes:
      testvolume:
        mount: "/mnt/data"
        seed: "workload_seed"
        filesystem: "ext4"
    

    Where

    • API_KEY: Is the docker registry access token.
    • COMPOSE_VALUE: is the base64 encode of file compose.tgz.
  • Prepare the env.yaml, the following is an example:

    type: env
    logging:
      logDNA:
        ingestionKey: ${INGESTIONKEY}
        hostname: ${LOGDNA_ENDPOINT}
        port: ${LOGDNA_PORT}
    volumes:
      testvolume:
        seed: "env_seed"
    

    Where

    • INGESTIONKEY: Is the logDNA Ingestion key value.
    • LOGDNA_ENDPOINT: Is the logDNA service instance hostname, example: syslog-a.us-south.logging.cloud.ibm.com.
    • LOGDNA_PORT: Is the logDNA service port, example: 6514
  • Generate an encrypted contract. CONTRACT_KEY is the file path of the encryption certificate, you can download the encryption certificate and extracting the public key.

    WORKLOAD="$(pwd)/workload.yaml"
    CONTRACT_KEY=${CONTRACT_KEY}
    PASSWORD="$(openssl rand 32 | base64 -w0)"
    ENCRYPTED_PASSWORD="$(echo -n "$PASSWORD" | base64 -d | openssl rsautl -encrypt -inkey $CONTRACT_KEY -certin | base64 -w0 )"
    ENCRYPTED_WORKLOAD="$(echo -n "$PASSWORD" | base64 -d | openssl enc -aes-256-cbc -pbkdf2 -pass stdin -in "$WORKLOAD" | base64 -w0)"
    echo "workload: \"hyper-protect-basic.${ENCRYPTED_PASSWORD}.${ENCRYPTED_WORKLOAD}\"" > contract_encrypt.txt
    ENV="$(pwd)/env.yaml"
    CONTRACT_KEY=${CONTRACT_KEY}
    PASSWORD="$(openssl rand 32 | base64 -w0)"
    ENCRYPTED_PASSWORD="$(echo -n "$PASSWORD" | base64 -d | openssl rsautl -encrypt -inkey $CONTRACT_KEY  -certin | base64 -w0)"
    ENCRYPTED_ENV="$(echo -n "$PASSWORD" | base64 -d | openssl enc -aes-256-cbc -pbkdf2 -pass stdin -in "$ENV" | base64 -w0)"
    echo "env: \"hyper-protect-basic.${ENCRYPTED_PASSWORD}.${ENCRYPTED_ENV}\"" >> contract_encrypt.txt
    

4.3.4 Create a Secure Execution VM

You can create a Secure Execution VM either on Hyper Protect Virtual Servers on premises or on Hyper Protect Virtual Servers for IBM Virtual Private Cloud (VPC).

4.3.5 Verify GREP11 server deployment

You can use the nmap tool to verify the on premises or on Cloud deployment by checking the port status of GREP1 server. If the port status is open, the grep11server runs normally. For example,

nmap -A -T4 -p 9876 <SE_VM_IP>
For example:
    nmap -A -T4 -p 9876 169.48.158.229
    nmap -A -T4 -p 9876 150.238.54.207
$ nmap -A -T4 -p 9876 169.48.158.229
Starting Nmap 7.92 ( https://nmap.org ) at 2022-08-06 08:44 CST
Nmap scan report for 169.48.158.229
Host is up (0.19s latency).
PORT     STATE SERVICE VERSION
9876/tcp open  ssl/sd?
| ssl-cert: Subject: commonName=169.48.158.229/organizationName=IBM/stateOrProvinceName=California/countryName=US
| Subject Alternative Name: DNS:127.0.0.1:9876, DNS:localhost:9876, DNS:9.47.152.236:9876, DNS:9.47.152.179:9876, DNS:150.238.54.207:9876, DNS:169.48.158.229:9876, IP Address:127.0.0.1, IP Address:169.48.158.229, IP Address:9.47.152.236, IP Address:9.47.152.179, IP Address:150.238.54.207
| Not valid before: 2022-08-04T01:11:10
|_Not valid after:  2024-08-03T01:11:10
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 44.30 seconds

4.3.6 View logs from logDNA instance console

Open the console of the configured logDNA instance and monitor realtime logs to check status of the Secure Execution VM.

No special capability is required by the GREP11 server container.

You can view the logs of GREP11, which the client for the Crypto Express Network API for Secure Execution Enclaves server, from the container logs. GREP11 logs can be configured to be collected by external log server, for example, the Rsyslog server.

5. Configure GREP11 client with enabled mTLS authentication

If you want to access cloud HSM on IBM Cloud Hyper Protect Crypto Services remotely to perform cryptographic operations, you need to generate a GREP11 API request in your on-prem environment to enable mutual TLS authentication for GREP11 API instead of IBM Cloud IAM authentication.

Prepare the following mutual TLS client certificates for connection with GREP11 server:

Filename Description
grep11-ca.pem GREP11 server CA certificates (Used as CA root on GREP11server side in mTLS connection.)
grep11-client.key GREP11 client private key
grep11-client.pem GREP11 client public certificate

Add the following snippet into your code:

cert, _ := tls.LoadX509KeyPair("grep11-client.pem ", "grep11-client.key")
var callOpts = []grpc.DialOption{
  grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{Certificates: []tls.Certificate{cert}}))
}

Now you can perform cryptographic operations by remotely accessing the functions with the Enterprise PKCS #11 (EP11) over gRPC API calls. For more details, see Cryptographic operations: GREP11 API.