Populating deployment files

Edit online

To deploy the Z APM Connect DG, populate the deployment files. If you are using Kubernetes, you can complete this task either by using Helm to create a Helm values file if Helm is installed, or by manually editing yaml manifest files and applying them to the cluster. If you are using OpenShift, you must use Helm to configure the values.yaml file. Helm is an industry standard tool that will help simplify the configuration, deployment, and maintenance of the Distributed Gateway deployment. You can also find the recommendations for pod limits and replicas on this page.

Note: If you're using fix pack 2 or earlier, follow the instructions in FP2/zapm_dg_openshift_values.html.

Option 1: Using Helm to configure a values.yaml file

Before you begin

Make sure that Helm V3.1 or later is installed. For more information about installing Helm, see Installing Helm.

About this task

Helm uses values files to fill in the template yamls. Before installing the Helm Chart for Z APM Connect DG, tell Helm about the desired configuration.

The values.yaml file needs to include the image repository to use, Instana details, the names of the Secrets created in Pre-deployment configuration, and other settings. A sample value.yaml is included with values.yaml.

Procedure

For production, the sample values.yaml is located at ./production/helm-deployment/values.yaml. It is recommended to create a backup copy before making edits.

In order to deploy the Distributed Gateway, you need to configure the parameters according to your needs. See the following example of which parameters are typically configured.
Tip: General tips when editing values.yaml
  • Spaces are important in yaml file. Each indentation needs to be two spaces.
  • It is highly recommended to use an editor that allows you to easily see spaces and search easily. A code editor with a yaml extension is recommended.

– Global values

Global values are located in the beginning of the sample values.yaml file and apply to all images.

Navigate to the following Values Section: 
# Global values
clusterType:

default:
Refer to the following table of parameter description to provide the configuration details relevant to your deployment.
Table 1. Configuration parameters for global values section of values.yaml file
Parameter Description Options
clusterType Tells Helm whether or not to enable cluster specific APIs
  • openshift - routes must be created for each endpoint.
  • kubernetes - routes do not need to be created.
imageRepository Points to an image registry where the Z APM Connect DG images are stored. These images will be pulled automatically by Helm. <user input>
imagePullPolicy Tells OpenShift/Kubernetes when to pull an image and when to use a cached image.
  • Always - always pull the image.
  • IfNotPresent - use cached images to save time.
  • Never - never pull the image.
imagePullSecrets

Name of OpenShift/Kubernetes secret used for authenticating with an external private registry where images will be pulled from.

 regcred
resources (Optional) Default resources section allows you to define limits and requirements for CPU and memory for each component.

However, it's not recommended to use the same resource allocation for all components. Instead, it's better to adjust resources individually for each component. See Table 11 to allocate specific resources for each component.

If no limits or requests are desired, the resources section can be left out. For more information about resource limits and requests, see Kubernetes documentation.

 The following example shows both a limit and request of 100 millicpus (⅒ of a core) and 128 megabytes that will be applied to each pod of the Z APM Connect Distributed Gateway: 
default:
...
  resources:
    limits:
      cpu: 100m
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi

– Transaction Processor Values

Navigate to the following Values Section: 
statefulsets:
  transactionProcessor:
Table 2. Configuration parameters for transactionProcessor section of values.yaml file
Parameter Description Options
tlsEnabled Enables/Disables TLS between z/OS and the connection-manager component of the Distributed Gateway.
  • If using OpenShift, TLS needs to be enabled by default.
  • If using plaintext connections (not recommended) or TLS termination with a proxy, TLS should not be enabled.
  • true
  • false
Note: If set to true, the transactionProcessor:secretName field must be specified.
secretName (Optional) Name of the secret containing the truststore.jks and keystore.jks for authentication with the connection manager component. <user input>
Important: If TLS is enabled, the field must match the Connection Manager secret name created in the previous step: Pre-deployment configuration. For example, if you follow the given example in the previous step, the field should be ingress-auth.
advertisedHostname FQDN (Fully Qualified Domain Name) of either the master node of the Kubernetes cluster, or the route that is created for OpenShift.
Note: When using OpenShift, this value will be used when configuring Z APM Connect Base TTServerString parameter to connect to the Distributed Gateway.
 <user input>
Tip: For OpenShift, the route looks like ingress.apps.openshift.company.domain. For example, if the OpenShift console listens at console-openshift-console.apps.my.company.com, this advertisedHostname should be ingress.apps.my.company.com.
advertisedPort Port used to contact the connection manager component externally. Used for testing the connection.
  • 443 - if using OpenShift
  • 30455 - if using Kubernetes
deployments.cdp-factory.smf_delimiter (Optional) Character(s) to enable custom Db2 SMF delimiter. <user input>
Tip: Any non-alphanumeric character is acceptable. For instance, characters like "=" or "~" are considered valid, as long as the chosen character matches the db2CorrelationDelimiter configured at the Instana agent level. See Configuring Instana Agent for how db2CorrelationDelimiter is set if a custom delimiter is desired.
logLevel Logging level of all connection-manager, event-partitioner, span-factory, span collector, transaction-factory, and cpd-factory.
  • OFF
  • FATAL
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • TRACE
  • ALL
redisSecret (Optional) Name of the secret containing your Redis Database password. <user input>
Important: The field must match the Redis Database secret name created in the previous step: Pre-deployment configuration. For example, if you follow the given example in the previous step, the field should be redis-auth.
Note: If using a proxy

If a proxy such as HAProxy or Nginx is used to handle connections, make sure that the advertisedHostname fields match the desired hostnames of the proxy routes. The advertisedHostname fields should match whichever port the proxy will be listening for traffic on externally, and the proxy should be configured to contact Connection Manager on port 30455.

– Instana-Exporter Values

Navigate to the following Values Section: 
statefulsets:
  instanaExporter:
    servers:
The instanaExporter section describes connection information for Instana servers to be used by Z APM Connect Distributed Gateway. Each parameter is as described below:
Table 3. Configuration parameters for instanaExporter section of values.yaml file
Parameter Description Options
name Name of the Instana backend – used for differentiation only (arbitrary but must be lower case) <user input>
hostEndpoint INSTANA_ENDPOINT_URL for the serverless ingestion for the Instana server. See Instana server for details. <user input>
agentKey INSTANA_AGENT_KEY for the Instana server. See Instana server for details. <user input>
deployment Type of Instana server.
  • saas
  • self-hosted
secretName (Optional) Name of the secret that contains the certificate signed by an internal CA or self-signed, if one is used. Commonly used for On-Prem Instana servers. <user input>
Important: If TLS is enabled, the field must match the Instana secret name created in the previous step: Pre-deployment configuration. For example, if you follow the given example in the previous step, the field should be instana-auth.
certName (Optional) Name of certificate used inside the secretName, if one is used. Commonly used for On-Prem Instana servers. <user input>
proxy.host (Optional) Hostname of forward proxy to use for connecting to an Instana server <user input>
proxy:port (Optional) Port on which the forward proxy listens to connect with an AppDynamics controller. <user input>

– Kafka Values

Navigate to the following Values Section: 
deployments:
  kafka:
Table 4. Configuration parameters for Kafka section of values.yaml file
Parameter Description Options
advertisedHostname FQDN (Fully Qualified Domain Name) of either the master node of the Kubernetes cluster, or the route that is created for OpenShift. <user input>
Tip: For OpenShift, the route looks like kafka.apps.openshift.company.domain. For example, if the OpenShift console listens at console-openshift-console.apps.my.company.com, this advertisedHostname should be kafka.apps.my.company.com.
advertisedPort Port used to contact the Kafka component externally.
  • 443 - if using OpenShift
  • 30090 - if using Kubernetes
externalTlsEnabled Enables TLS two-way authentication for any Kafka clients connecting using the advertised hostname.
  • If using Openshift, TLS needs to be enabled by default.
  • If using plaintext connections (not recommended) or TLS termination with a proxy, TLS should not be enabled.
  • true
  • false
Note: If set to true, the kafka:secretName field must be specified.
internalTlsEnabled Enables TLS two-way authentication for the DG pods running inside the cluster. Enable this field if secure communication within your cluster is desired.
Note: Enabling internal TLS will result in increased performance overhead costs, specifically in required CPU usage.
  • true
  • false
Note: If set to true, the kafka:secretName field must be specified.
secretName (Optional) Name of the secret containing the truststore.jks and keystore.jks for authentication with the Kafka component. <user input>
Important: If TLS is enabled, the field must match the kafka secret name specified in the previous step: Pre-deployment configuration. For example, if you follow the given example to create the kafka secret in the previous step, it should be kafka-auth.
Note: If using a proxy

If a proxy such as HAProxy or Nginx is used to handle connections, make sure that the advertisedHostname fields match the desired hostnames of the proxy routes. The advertisedHostname fields should match whichever port the proxy will be listening for traffic on externally, and the proxy should be configured to contact Kafka on port 30090.

* Refer to Deploying Z APM Connect DG for production in a cluster information about where to find this value.

– Redis Values

Navigate to the following Values Section: 
deployments:
  redis:
Table 5. Configuration parameters for Redis section of values.yaml file
Parameter Description Options
redisSecret (Optional) Name of the secret containing your Redis Database password. <user input>
Important: The field must match the Redis Database secret name created in the previous step: Pre-deployment configuration. For example, if you follow the given example in the previous step, the field should be redis-auth.

Option 2: Filling in Kubernetes manifest files

As an alternative to Helm, you can manually edit yaml manifest files and apply them to the cluster.
Restriction: If you are using OpenShift, this option does not apply.
A number of manifest yaml files are provided under the production/manifests-deployments/ directory where you extracted the installation package. Each of these files must be updated before you apply them to the cluster. A series of <placeholder> fields need to be replaced with the proper value in each of the following manifest yaml files.
├── manifests-deployments
│  ├── instana-exporter.yaml
│  ├── kafka.yaml
│  ├── redis.yaml
│  ├── transaction-processor.yaml
Table 6. kafka.yaml
Placeholders Description
<repo-where-images-live> Image repository where the images were loaded.
<advertised listening name for Kafka> Advertised listening name for Kafka connections, z/OS Connect EE or CICS Transaction Gateway.
<advertised listening port for Kafka> Advertised listening port for Kafka.
<kafka-auth> (Kafka) Name of secret where kafka.keystore.jks and kafka.truststore.jks and their passwords are set.
<SSL or PLAINTEXT> Enables TLS two-way authentication for the DG pods running inside the cluster. Enable this field if secure communication within your cluster is desired.
Note: Enabling internal TLS will result in increased performance overhead costs, specifically in required CPU usage.
Table 7. redis.yaml
Placeholders Description
<repo-where-images-live> Image repository where the images were loaded.
<redis-auth> Name of the secret containing your Redis Database password.
Table 8. instana.exporter.yaml
Placeholders Description
<repo-where-images-live> Image repository where the images were loaded.
<instana endpoint url> INSTANA_ENDPOINT_URL as listed in the following steps:
    1. Sign in to Instana and click ...More > Agents > Install Agents > AWS.
    2. From the Technology list, select AWS Lambda.
<instana agent key> INSTANA_AGENT_KEY as listed in the following steps:
    1. Sign in to Instana and click ...More > Agents > Install Agents > AWS.
    2. From the Technology list, select AWS Lambda.
<true or false> true if using a SaaS controller, false otherwise. Search for ZAPM_INSTANA_IS_SAAS.
<instana-auth> Secret that contains the certificate that is signed by an internal CA, if one is used.
<cert file name inside the self signed certificate secret> If the Instana server uses a self-signed certificate (not recommended) or the certificate is signed by a private CA, set to the name of the signing certificate that is contained in the secret and uncomment the volumes and volumeMounts sections of the specific instana-exporter deployment, otherwise set to an empty string ("").
<hostname or IP of proxy machine> Hostname or IP of proxy machine, if present. Otherwise, this value and its corresponding name field must be commented out (placing a # in front of the line).
<1234> Port of proxy machine, if present. Otherwise, this value and its corresponding name field must be commented out (place a # in front of the line). Search for ZAPM_INSTANA_PROXY_PORT.
<kafka-auth> (Kafka) Name of secret where kafka.keystore.jks and kafka.truststore.jks and their passwords are set.
<required or none> Enables TLS two-way authentication for the DG pods running inside the cluster. Enable this field if secure communication within your cluster is desired.
Note: Enabling internal TLS will result in increased performance overhead costs, specifically in required CPU usage.
Table 9. transaction-processor.yaml
Placeholders Description
<repo-where-images-live> Image repository where the images were loaded.
<ingress-auth> (Z APM Connect Base listener) Name of secret where keystore.jks and truststore.jks and their passwords are set.
<redis-auth> Name of the secret containing your Redis Database password.
<kafka-auth> (Kafka) Name of secret where kafka.keystore.jks and kafka.truststore.jks and their passwords are set.
<required or none> Enables TLS two-way authentication for the DG pods running inside the cluster. Enable this field if secure communication within your cluster is desired.
Note: Enabling internal TLS will result in increased performance overhead costs, specifically in required CPU usage.

Recommendations for Pod Limits and Replicas

It is recommended that you set pod limits and replicas for the deployed Z APM Connect components. The recommended values for replicas and resource limits are based on the performance test results. Refer to the table that matches your anticipated workload.

  • If the transactions are no more than 15k transactions per second, follow the recommended values in table 10.
  • If the transactions are between 15k to 30k per second, follow the recommended values in table 11.
  • Workloads that exceed 30k transactions per second may require an increase in replicas and resource limits to accommodate the higher transactions per second.
Note: Performance tests for Z APM Connect Distributed Gateway were conducted on an OpenShift cluster with 8 worker nodes. Each worker node was an x86 virtual machine with 16 cores and 25GB of RAM. The tests use simple traces that include a parent transaction and 2 to 3 child transactions.

The recommended values may require adjustments depending on the computing environment, including factors such as hardware specifications, operating system, and application workloads. For example, more complex transaction data will result in larger event sizes and may require adjustments.

Table 10. Suggested pod limits and replicas for each deployed component with ≤ 15k Transactions/second
Deployment Request CPU (Min) Request RAM (Min) Limit CPU (Max) Limit RAM (Max) Minimum Replicas
connection-manager 200m 300Mi 200m 300Mi 3
event-partitioner 700m 700Mi 700m 700Mi 3
span-factory 1000m 2Gi 1000m 2Gi 5
span-collector 500m 300Mi 500m 300Mi 5
transaction-factory 500m 800Mi 500m 800Mi 5
instana-exporter 600m 1500Mi 600m 1500Mi 5
kafka 1500m 8Gi 1500m 8Gi 1
redis 4000m 7Gi 4000m 7Gi 1
Table 11. Suggested pod limits and replicas for each deployed component with 15k-30k Transactions/second
Deployment Request CPU (Min) Request RAM (Min) Limit CPU (Max) Limit RAM (Max) Minimum Replicas
connection-manager 200m 300Mi 200m 300Mi 6
event-partitioner 700m 1Gi 700m 1Gi 7
span-factory 1000m 2Gi 1000m 2Gi 15
span-collector 500m 1Gi 500m 1Gi 5
transaction-factory 500m 1Gi 500m 1Gi 5
instana-exporter 1000m 2Gi 1000m 2Gi 15
kafka 8000m 23Gi 8000m 23Gi 1
redis 4000m 7Gi 4000m 7Gi 1

Kubernetes requests represent the minimum resources that a pod can have, while Kubernetes limits define the maximum resources. CPU resources are measured in millicores, where 1000m is equivalent to 1 CPU core. Memory is measured in bytes and expressed as mebibyte values, which are similar to megabytes.

Attention: For scaling purposes, you need to have at least one Z APM Connect Base Courier for each Connection Manager replica. For example, if you scale connection-manager to 4 replicas, you would need 4 Z APM Connect Base Couriers in order to fully utilize all 4 connection-manager replicas.
Attention: If you enable Internal Kafka TLS two-way authentication for your Distributed Gateway deployment, there will be increased performance overhead which will result in increased resource requirements.
Tip: To update replicas, modify the "replicas" parameter in the values.yaml file for Helm. For Kubernetes manifest files, update transaction-processor.yaml and instana-exporter.yaml.