Certificates in a Kubernetes environment

Use of cert-manager is recommended for managing certificates in a Kubernetes environment.

When deploying on Kubernetes, choose one of the following methods for creating internal certificates:

  • Do not specify any internal certificates. In this case, the operator generates all of the internal certificates without using a certificate manager. The CR definition to enable this option is:
    microServiceSecurity: custom
  • Specify some, or all, of the internal certificates yourself without using a certificate manager. In this case, any certificates that you do not create are created by the operator. You retain control over the certificates that you create. The CR definition to enable this option is:
    microServiceSecurity: custom
  • Configure cert-manager to generate the internal certificates. In this case, cert-manager creates and manages the certificates. The CR definition to enable this option is:
    microServiceSecurity: certManager

    To use cert-manager for generating the certificates, download and install cert-manager v1.11.5 from https://github.com/cert-manager/cert-manager/releases/tag/v1.11.5 and create an Issuer CR in the same namespace where you will install API Connect; for example:

    apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigning-issuer
spec:
  selfSigned: {}

    In the CR for the API Connect Management subsystem, specify a certManagerIssuer. The certManagerIssuer can be configured to be a self-signing issuer, as in the following example:

    
microServiceSecurity: certManager
certManagerIssuer:
  name: selfsigning-issuer
  kind: Issuer

    API Connect supports any other kind of Issuer supported by cert-manager, in addition to the self signed type.

In all the CRs, certs are expected at the level of any endpoint. For example, for the cloud admin endpoint of the Management subsystem you could have:


cloudManagerEndpoint:
    annotations:
      cert-manager.io/issuer: ingress-issuer
    hosts: 
    - name: admin.example.com
      secretName: cm-endpoint

The previous entry lets cert-manager automatically generate a cm-endpoint TLS secret using the ingress-issuer. One way to define the ingress-issuer is as follows:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigning-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-ca
spec:
  secretName: ingress-ca
  commonName: "ingress-ca"
  usages:
  - digital signature
  - key encipherment
  - cert sign
  isCA: true
  duration: 87600h # 10 years
  renewBefore: 720h # 30 days
  privateKey:
    rotationPolicy: Always
  issuerRef:
    name: selfsigning-issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-issuer
spec:
  ca:
    secretName: ingress-ca

Another way to configure the cloud admin endpoint would be with:

 cloudManagerEndpoint:
    hosts: 
    - name: admin.example.com
      secretName: my-cm-endpoint

Where my-cm-endpoint is the name of a secret that you previously created, and is present in the same namespace as the subsystem, and contains TLS certificate files for termination of the endpoint.

Creating secrets and generating certificates

There is a number of "secrets" for which you can provide a name in the CRs:

  • Portal admin secret. 
    portal:
  admin:
    secretName:
  • Analytics ingestion secret. 
    analytics:
  ingestion:
    secretName: analytics-ingestion-client
  • APIC Gateway Service TLS secret. Template entry from gateway_cr_v6.yaml (in the file name, "v6" refers to the API Gateway service):
    
  apicGatewayServiceTLS:
    secretName:
  • APIC Gateway Peering TLS secret. Template entry from gateway_cr_v6.yaml (in the file name, "v6" refers to the API Gateway service): 
    
  apicGatewayPeeringTLS:
    secretName:

You can create manually create secrets that contain the tls.crt, tls.key, and tls.ca files for a certificate. You can then refer to them by name in their subsystem CR. However, a simpler way to have those secrets created is by using cert-manager. The following example shows how to generate those certificates with cert-manager:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigning-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-ca
spec:
  secretName: ingress-ca
  commonName: "ingress-ca"
  usages:
  - digital signature
  - key encipherment
  - cert sign
  isCA: true
  duration: 87600h # 10 years
  renewBefore: 720h # 30 days
  privateKey:
    rotationPolicy: Always
  issuerRef:
    name: selfsigning-issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ingress-issuer
spec:
  ca:
    secretName: ingress-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: portal-admin-client
spec:
  commonName: portal-admin-client
  secretName: portal-admin-client
  issuerRef:
    name: ingress-issuer
  usages:
  - "client auth"
  - "signing"
  - "key encipherment"
  duration: 17520h # 2 years
  renewBefore: 720h # 30 days
  privateKey:
    rotationPolicy: Always
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: analytics-ingestion-client
spec:
  commonName: analytics-ingestion-client
  secretName: analytics-ingestion-client
  issuerRef:
    name: ingress-issuer
  usages:
  - "client auth"
  - "signing"
  - "key encipherment"
  duration: 17520h # 2 years
  renewBefore: 720h # 30 days
  privateKey:
    rotationPolicy: Always
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gateway-service
spec:
  commonName: gateway-service
  secretName: gateway-service
  issuerRef:
    name: ingress-issuer
  usages:
  - "client auth"
  - "signing"
  - "key encipherment"
  duration: 17520h # 2 years
  renewBefore: 720h # 30 days
  privateKey:
    rotationPolicy: Always
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gateway-peering
spec:
  commonName: gateway-peering
  secretName: gateway-peering
  issuerRef:
    name: ingress-issuer
  usages:
  - "client auth"
  - "signing"
  - "key encipherment"
  duration: 17520h # 2 years
  renewBefore: 720h # 30 days
  privateKey:
    rotationPolicy: Always

So with the previous example, in the subsystem CRs (for Management and Gateway) the following values could be set:

  • portal:
  admin:
    secretName: portal-admin-client
  • analytics:
  ingestion:
    secretName: analytics-ingestion-client
  • apicGatewayServiceTLS:
  secretName: gateway-service
  • apicGatewayPeeringTLS:
  secretName: gateway-peering

Also, the Client certificate should be signed by same CA as the one for the endpoint certificates, and the server side should be configured with the expected Subject DN for the client. For example:

  • The portalAdminSecret certificate and the certificate used to terminate portalAdminEndpoint must have the same CA (in the example the CA is ingress-ca for both). Also, the adminClientSubjectDN in the Portal subsystem CR should be set to the Subject DN of portal.admin.secretName. For example:
    adminClientSubjectDN: CN=portal-admin-client,O=cert-manager
  • Similarly, the analyticsIngestionSecret and the Analytics ingestion endpoint certificate should share the same CA. The ingestion.clientSubjectDN should be set to the Subject DN of the analytics.ingestion.secretName certificate. 
    
  ingestion:
    endpoint:
      annotations:
        cert-manager.io/issuer: ingress-issuer
      hosts: 
      - name: ai.example.com
        secretName: analytics-ai-endpoint
    clientSubjectDN: CN=analytics-ingestion-client,O=cert-manager
Important: When installing more than one Analytics subsystem in the same Kubernetes namespace, all secret names must be unique.

Creating TLS secret using openssl

Use of the cert-manager is recommended for managing certificates. However, you can choose not to use cert-manager, and instead create certificates manually. Requirements for manually created certificates:

  • Extended Key Usage (EKU), either serverAuth or clientAuth depending upon the type of certificate. Certificates of type Server must have an Extended Key Usage with serverAuth purpose. Certificates of type Client must have an Extended Key Usage with clientAuth purpose.
  • Subject Alternative Name (SAN) for the required hosts
  • If a certificate is signed by an internal or custom CA, include the full chain in the end certificate. If you omit the full chain, then a user who uses openSSL to access the endpoint will see the following error: error:num=20:unable to get local issuer certificate
    The following example shows the full chain for an end certificate (signed by a custom CA):
    -------BEGIN CERTIFICATE -----
Cert contents for end-cert
--------END CERTIFICATE ----------
-------BEGIN CERTIFICATE -----
Cert contents for Intermediate-CA
--------END CERTIFICATE ----------
-------BEGIN CERTIFICATE -----
Cert contents for Root CA
--------END CERTIFICATE ----------

The following steps can be used as an alternative to using cert-manager. If you choose to create your certificates this way, you lose some of the management features of cert-manager. For example, you must manually update your certificates when their expiration date gets near.

  1. Generate the custom certificate with the appropriate EKU and SAN. You will need to obtain the private key, public certificate, and CA certificates in non-password-protected PEM format for the custom certificate. Following is an example for how to generate a certificate (platform-api-example) with an EKU serverAuth and SAN using openssl:
    openssl x509 -req -days 360 -in platform-api-example.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -out platform-api-cert -sha256  
-extfile <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=DNS:fqdn.myserver.com\nextendedKeyUsage=serverAuth")) 
-extensions SAN
    where
    • DNS:fqdn.myserver.com is the fully qualified domain name of the endpoint the certificate applies to.
    • platform-api-example.csr is the file name for the certificate signing request

    Following is an example for how to generate a certificate (portal-client) with an EKU clientAuth and SAN using openssl:

    openssl x509 -req -days 360 -in portal-client-example.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -out portal-client-cert  
-sha256 -extfile <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nkeyUsage=critical, digitalSignature,keyEncipherment\nextendedKeyUsage = clientAuth\nbasicConstraints=critical, 
CA:FALSE\nsubjectKeyIdentifier=hash\n")) -extensions SAN
  2. Create a Kubernetes secret containing tls.cert, tls.key, and ca.cert.

Setting an encryption-secret for the management database using openssl

You can optionally set the encryption-secret for the management database. If you do not set one, one is automatically generated for you.

  1. The encryption-secret is a secure random bytes password used for field level encryption in the management database. You can generate 128 random bytes using the following command in openssl:
    openssl rand -out /path/to/secret/encryption-secret.bin 128
  2. Use kubectl to put the generated secret into management-encryption-key:
    kubectl create secret generic management-encryption-key -n <namespace>
      --from-file=/path/to/secret/encryption_secret.bin

    where <namespace> is the namespace where the Management CR is going to be created)

  3. Make sure that in management_cr.yaml, encryptionSecret.secretName points to the name of the secret created in the previous step.
Important: Back up the encryption-secret.bin file to a safe location. If this file lost it will not be possible to access the management database or its backups.