Deploying a GREP11 server as a Hyper Protect Container Runtime instance
Complete the following steps to integrate with GREP11 server.
- Prepare GREP11 server image
- Prepare configuration files for c16 client
- Prepare configuration files for GREP11 server
- Deploy GREP11 server
- Configure GREP11 client with enabled mTLS authentication
1. Prepare GREP11 server image
Load the GREP11 server image from the tar
archive. If you don't have the tar
archive, see Downloading Crypto Express Network API for Secure Execution Enclaves.
Record the tag of the image, which is required to construct docker-compose.yml
in Step 4.1.
docker load -i grep11server_s390x_1.1.2.1.tar
The following snippet shows an example of the output:
Loaded image: sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.1.2
2. Prepare configuration files for c16 client
To prepare and collect the following files that are required for connecting the c16 client to the c16 server, see Generate server certificate (openssl) with the generated server CSR to be assigned by CA ROOT. Use these values to prepare the configuration file.
Filename | Description |
---|---|
c16client.yaml | c16 client configuration file |
c16server-ca.pem | c16server CA root certificate (used as CA root on client side in mTLS connection with c16server) |
c16server-client.key | c16server client private key (used as client key on client side in mTLS connection with c16server) |
c16server-client.pem | c16server client certificates (used as client certificate on client side in mTLS connection with c16server) |
c16server-restricted-server.pem | c16server server certificate (used to do restricted server certificate validation, which will ensure only the permitted target server (which is the same as the one used on the c16server side) can be accessed) |
Sample of c16client.yaml
as below, and variables are required to replace:
#trace, debug, info, warn, err, error, critical, off
loglevel: 'debug'
servers:
- hostname: ${C16SERVER_IP}
port: 9001
mTLS: true
server_cert_file: "/cfg/c16server-ca.pem"
client_key_file: "/cfg/c16server-client.key"
client_cert_file: "/cfg/c16server-client.pem"
Where C16SERVER_IP: Is the IP address of the c16 server.
Note:
- If no
restrict_server_cert_file
parameter is set or it is empty, then c16client will work normally. - If you want to restrict the server accessibility, set the parameter
restrict_server_cert_file
to the right c16server certificate.
Trouble Shooting: If restrict_server_cert_file
is set, and you see the following error message from the server:
grep11server-ep11server-1 | [c16client][error] c16client::c16OpenAdapter: Failed to verify the identity of target server, please re-config and connect with permitted server.
Ensure that you do the following:
- Set the parameter
restrict_server_cert_file
with the correct c16server certificate. - Connect to the target c16server.
3. Prepare configuration files for GREP11 server
To prepare and collect the following certificates that are required for configuring the GREP11 server, see Creating OpenSSL certificates for GREP11 Servers.
Filename | Description |
---|---|
grep11server.yaml | GREP11 server configuration file |
grep11-ca.pem | GREP11 server CA certificates(used as CA root on GREP11server side in mTLS connection) |
grep11-server.key | GREP11 server private key(used as server key on GREP11server side in mTLS connection) |
grep11-server.pem | GREP11 server certificates(used as server certificate on GREP11server side in mTLS connection) |
HSM domain values of c16server can be acquired from this step Configuring Crypto Express Card Domain.
Sample of grep11server.yaml
:
#
# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#
logging:
# Package log levels
# Levels: info, warning, debug, error, fatal, trace, and panic
levels:
# example below
# entry: debug
###################################
## GRPC SERVICES TALKING TO GREP11 server ##
ep11crypto:
enabled: true
connection:
address: 0.0.0.0
port: 9876
# Secure connection TLS options
tls:
enabled: true
# certfile, keyfilie and cacert refer to the pem files that holds the certs
certfile: /cfg/grep11-server.pem
keyfile: /cfg/grep11-server.key
mutual: true
cacert: /cfg/grep11-ca.pem
# same as above, but instead of filename, contents of PEM can be in an environment
# variable (i.e. to avoid mounting files into docker container)
cacertbytes:
certfilebytes:
keyfilebytes:
# Server TCP/IP connection monitoring
# serverKeepaliveTime is the duration in seconds after which if the server
# does not see any activity from the client it pings the client to see
# if it is alive
# serverKeepaliveTimeout is the duration the server waits for a response
# from the client after sending a ping before closing the connection
keepalive:
serverKeepaliveTime: 30
serverKeepaliveTimeout: 5
# Comma-separated list of card.domain tuples. card and domain are hex numbers
# i.e. "01.a9,c.10" correspods to 169th domain on first card and 16th domain on 12th card
domain: "${C16SERVER_HSM_DOMAIN}"
Where C16SERVER_HSM_DOMAIN: HSM domains, like "00.003a,01.003a"
, are configured on the c16server, maximum 2 domains are supported.
4. Deploy GREP11 server
GREP11 server can be deployed either on a local s390x environment or as a Hyper Protect Container Runtime instance.
4.1 Prepare docker-compose.yml
Prepare docker-compose.yml
, which is used to boot up the GREP11 server container.
Note: You must change the image tag to grep11server image
Sample of docker-compose.yml
:
services:
ep11server:
user: "0"
image: ${GREP11SERVER_IMAGE_TAG}
ports:
- 9876:9876
volumes:
- ./c16client.yaml:/etc/c16/c16client.yaml
- ./c16-ca.pem:/cfg/c16server-ca.pem
- ./c16-client.key:/cfg/c16server-client.key
- ./c16-client.pem:/cfg/c16server-client.pem
- ./grep11server.yaml:/etc/ep11server/ep11server.yaml
- ./grep11-ca.pem:/cfg/grep11-ca.pem
- ./grep11-server.pem:/cfg/grep11-server.pem
- ./grep11-server.key:/cfg/grep11-server.key
Where GREP11SERVER_IMAGE_TAG is the grep11server image tag, like "us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1@sha256:54334c919a93c45f70d0252d8518d24b64255a88a073b35239f95d056d444ab8"
For single or multiple containers, you can also use the play
subsection. In the play
subsection, you can define the workload via Pod descriptors.
Each pod can contain one or more container definitions. Descriptors can be provided as shown in the following example:
apiVersion: v1
kind: Pod
metadata:
name: grep11server
spec:
securityContext:
privileged: true
containers:
- name: grep11server
image: us.icr.io/crypto-appliance-test/sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.0.0@sha256:7a6e20bd9c7aa5bc155212247859fcd5ee01d0005d123561e68fec3fa4452abe
volumeMounts:
- name: c16-client
mountPath: /etc/c16/c16client.yaml
- name: c16-ca
mountPath: /cfg/c16server-ca.pem
- name: c16-client-key
mountPath: /cfg/c16server-client.key
- name: c16-client-pem
mountPath: /cfg/c16server-client.pem
- name: grep11-config
mountPath: /etc/ep11server/ep11server.yaml
- name: grep11-ca
mountPath: /cfg/grep11-ca.pem
- name: grep11-server-pem
mountPath: /cfg/grep11-server.pem
- name: grep11-server-key
mountPath: /cfg/grep11-server.key
securityContext:
privileged: true
ports:
- containerPort: 9876
hostPort: 9876
volumes:
- name: c16-client
hostPath:
path: ./c16client.yaml
type: FileOrCreate
- name: c16-ca
hostPath:
path: ./c16-ca.pem
type: FileOrCreate
- name: c16-client-key
hostPath:
path: ./c16-client.key
type: FileOrCreate
- name: c16-client-pem
hostPath:
path: ./c16-client.pem
type: FileOrCreate
- name: grep11-config
hostPath:
path: ./grep11server.yaml
type: FileOrCreate
- name: grep11-ca
hostPath:
path: ./grep11-ca.pem
type: FileOrCreate
- name: grep11-server-pem
hostPath:
path: ./grep11-server.pem
type: FileOrCreate
- name: grep11-server-key
hostPath:
path: ./grep11-server.key
type: FileOrCreate
restartPolicy: Always
Run the following command to generate the base64 value of compose.tgz
:
tar -czvf compose.tgz docker-compose.yml c16client.yaml grep11server.yaml c16-ca.pem c16-client.pem c16-client.key grep11-ca.pem grep11-server.key grep11-server.pem
base64 -i compose.tgz | tr -d '\n' > compose.b64
Prepare the contract file as shown in the following example:
env: |
type: env
logging:
logDNA:
hostname: syslog-a.us-south.logging.cloud.ibm.com
ingestionKey: ab00e3....
port: 6514
volumes:
test0:
seed: stsolutiontest
workload: |
type: workload
play:
archive: ${COMPOSE_VALUE}
auths:
us.icr.io:
username: iamapikey
password: Eqx0TS....
volumes:
test0:
mount: /mnt/data0
seed: stsolutiontest
filesystem: ext4
4.2 Deploy GREP11 server container locally
You can deploy the GREP11 server container on a local s390x server by running the following command.
docker-compose up
4.3 Deploy GREP11 server as a Hyper Protect Container Runtime instance
A Secure Execution VM can be deployed on premises or on Cloud. Both of them use the same steps to prepare contract. For more information, see Hyper Protect Container Runtime.
Complete the following steps to deploy the GREP11 server:
- 4.3.1. Generate compose package
- 4.3.2. Create a logDNA instance
- 4.3.3. Prepare contract
- 4.3.4. Create Secure Execution VM
- 4.3.5. Verify GREP11 server deployment
- 4.3.6. View log from logDNA instance console
4.3.1 Generate compose package
Complete the following steps to generate the compose package.
-
Upload grep11server image to a docker registry
To build Secure Execution VM, it is required to upload the grep11server image to a docker registry. Both
us.icr.io
andhttps://index.docker.io/v1/
are supported docker registries. For example,API_KEY=<API-KEY> docker login --username iamapikey -p $API_KEY us.icr.io docker tag sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.1.2.1 us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1 docker push us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1 USERNAME=<USERNAME> PASSWORD=<PASSWORD> docker login --username $USERNAME -p $PASSWORD docker.io docker tag sys-hpcs-team-grep11-docker-local.artifactory.swg-devops.com/grep11-c16:grep11server_s390x-1.1.2.1 docker.io/hexiaohu/grep11server:s390x-1.1.2.1 docker push docker.io/hexiaohu/grep11server:s390x-1.1.2.1
-
Update the image tag in
docker-compose.yml
Update the image tag in
docker-compose.yml
to the value that reflects the image tag of docker registry. For example,... image: us.icr.io/crypto-appliance/grep11server:s390x-1.1.2.1@sha256:54334c919a93c45f70d0252d8518d24b64255a88a073b35239f95d056d444ab8 ...
-
Generate base64 value of
compose.tgz
Compact all of config files together with
docker-compose.yml
intocompose.tgz
and get the base64 value ofcompose.tgz
. Note: Ensure that the compose.tgz file contains only directories and regular files. Links or pipes are not supported.rm -f compose.tgz compose.b64 tar -czvf compose.tgz docker-compose.yml c16client.yaml grep11server.yaml c16server-ca.pem c16server-client.pem c16server-client.key grep11-ca.pem grep11-server.key grep11-server.pem base64 -i compose.tgz | tr -d '\n' > compose.b64
4.3.2 Create a logDNA instance
Create a logDNA instance on IBM Cloud and get the following access point value Ingestion Key
. For more information, see Getting started with Log Analysis and Getting ingestion keys.
Ingestion Key: <Ingestion Key Value>
Service Endponit: <Endpoint of service instance, eg. syslog-a.us-south.logging.cloud.ibm.com:6514>
4.3.3 Prepare contract
-
Prepare
workload.yaml
, the following is an example:Update the
auths
section to indicate the logon method of the docker registry, in which the grep11server image is located. Usingus.icr.io
as an example:type: workload auths: us.icr.io: password: ${API_KEY} username: iamapikey compose: archive: ${COMPOSE_VALUE} volumes: testvolume: mount: "/mnt/data" seed: "workload_seed" filesystem: "ext4"
Where
- API_KEY: Is the docker registry access token.
- COMPOSE_VALUE: is the base64 encode of file compose.tgz.
-
Prepare the
env.yaml
, the following is an example:type: env logging: logDNA: ingestionKey: ${INGESTIONKEY} hostname: ${LOGDNA_ENDPOINT} port: ${LOGDNA_PORT} volumes: testvolume: seed: "env_seed"
Where
- INGESTIONKEY: Is the logDNA Ingestion key value.
- LOGDNA_ENDPOINT: Is the logDNA service instance hostname, example: syslog-a.us-south.logging.cloud.ibm.com.
- LOGDNA_PORT: Is the logDNA service port, example: 6514
-
Generate an encrypted contract. CONTRACT_KEY is the file path of the encryption certificate, you can download the encryption certificate and extracting the public key.
WORKLOAD="$(pwd)/workload.yaml" CONTRACT_KEY=${CONTRACT_KEY} PASSWORD="$(openssl rand 32 | base64 -w0)" ENCRYPTED_PASSWORD="$(echo -n "$PASSWORD" | base64 -d | openssl rsautl -encrypt -inkey $CONTRACT_KEY -certin | base64 -w0 )" ENCRYPTED_WORKLOAD="$(echo -n "$PASSWORD" | base64 -d | openssl enc -aes-256-cbc -pbkdf2 -pass stdin -in "$WORKLOAD" | base64 -w0)" echo "workload: \"hyper-protect-basic.${ENCRYPTED_PASSWORD}.${ENCRYPTED_WORKLOAD}\"" > contract_encrypt.txt ENV="$(pwd)/env.yaml" CONTRACT_KEY=${CONTRACT_KEY} PASSWORD="$(openssl rand 32 | base64 -w0)" ENCRYPTED_PASSWORD="$(echo -n "$PASSWORD" | base64 -d | openssl rsautl -encrypt -inkey $CONTRACT_KEY -certin | base64 -w0)" ENCRYPTED_ENV="$(echo -n "$PASSWORD" | base64 -d | openssl enc -aes-256-cbc -pbkdf2 -pass stdin -in "$ENV" | base64 -w0)" echo "env: \"hyper-protect-basic.${ENCRYPTED_PASSWORD}.${ENCRYPTED_ENV}\"" >> contract_encrypt.txt
4.3.4 Create a Secure Execution VM
You can create a Secure Execution VM either on Hyper Protect Virtual Servers on premises or on Hyper Protect Virtual Servers for IBM Virtual Private Cloud (VPC).
-
On premises
Follow Setting up and configuring IBM Hyper Protect Virtual Servers to deploy GREP11 server as a Secure Execution VM on HPVS on-premises.
Notes:
User-Data
content should reference to the generated contract from Step 4.3.3 Prepare contract.- You must configure
iptables
to open required ports for GREP11 server, following step 11 in Setting up and configuring IBM Hyper Protect Virtual Servers.
-
On Cloud
Log in to IBM Cloud to create a Secure Execution VM. For more information, see Creating virtual server instances, and IBM Hyper Protect Container Runtime image.
Notes:
- Choose the generated contract from Step 4.3.3 Prepare contract.
- You must create a data volume and bind it with this VM. It can be a smallest size of 10GB.
- You must configure security policy group for the network VPC to open required ports.
4.3.5 Verify GREP11 server deployment
You can use the nmap
tool to verify the on premises or on Cloud deployment by checking the port status of GREP1 server. If the port status is open, the grep11server runs normally. For example,
nmap -A -T4 -p 9876 <SE_VM_IP>
For example:
nmap -A -T4 -p 9876 169.48.158.229
nmap -A -T4 -p 9876 150.238.54.207
$ nmap -A -T4 -p 9876 169.48.158.229
Starting Nmap 7.92 ( https://nmap.org ) at 2022-08-06 08:44 CST
Nmap scan report for 169.48.158.229
Host is up (0.19s latency).
PORT STATE SERVICE VERSION
9876/tcp open ssl/sd?
| ssl-cert: Subject: commonName=169.48.158.229/organizationName=IBM/stateOrProvinceName=California/countryName=US
| Subject Alternative Name: DNS:127.0.0.1:9876, DNS:localhost:9876, DNS:9.47.152.236:9876, DNS:9.47.152.179:9876, DNS:150.238.54.207:9876, DNS:169.48.158.229:9876, IP Address:127.0.0.1, IP Address:169.48.158.229, IP Address:9.47.152.236, IP Address:9.47.152.179, IP Address:150.238.54.207
| Not valid before: 2022-08-04T01:11:10
|_Not valid after: 2024-08-03T01:11:10
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 44.30 seconds
4.3.6 View logs from logDNA instance console
Open the console of the configured logDNA instance and monitor realtime logs to check status of the Secure Execution VM.
No special capability is required by the GREP11 server container.
You can view the logs of GREP11, which the client for the Crypto Express Network API for Secure Execution Enclaves server, from the container logs. GREP11 logs can be configured to be collected by external log server, for example, the Rsyslog server.
5. Configure GREP11 client with enabled mTLS authentication
If you want to access cloud HSM on IBM Cloud Hyper Protect Crypto Services remotely to perform cryptographic operations, you need to generate a GREP11 API request in your on-prem environment to enable mutual TLS authentication for GREP11 API instead of IBM Cloud IAM authentication.
Prepare the following mutual TLS client certificates for connection with GREP11 server:
Filename | Description |
---|---|
grep11-ca.pem |
GREP11 server CA certificates (Used as CA root on GREP11server side in mTLS connection.) |
grep11-client.key |
GREP11 client private key |
grep11-client.pem |
GREP11 client public certificate |
Add the following snippet into your code:
cert, _ := tls.LoadX509KeyPair("grep11-client.pem ", "grep11-client.key")
var callOpts = []grpc.DialOption{
grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{Certificates: []tls.Certificate{cert}}))
}
Now you can perform cryptographic operations by remotely accessing the functions with the Enterprise PKCS #11 (EP11) over gRPC API calls. For more details, see Cryptographic operations: GREP11 API.