External applications tracing data
Applications without an integrated IBM Cloud Pak for Integration tracing support can also push tracing data into the Operations Dashboard runtime. Such applications are called external applications in the Web Console.
Overview
In order to send and view tracing data from external applications, the following requirements must be met:
Operations Dashboard Web Console needs to be configured to display external applications tracing data. See Display system parameters for more information.
The external application needs to be deployed inside the IBM Cloud Pak for Integration cluster.
Operations Dashboard components must be integrated into the external application workload (see below).
Code changes needs to be made to the external application code (see below).
Registration process of the exernal application must be completed (see below).
Once the requirements are met, tracing data is sent to Operations Dashboard and is displayed in the Web Console:
External app overview dashboard is displayed.
External applications data is displayed as an additional capability in the "New traces by capability" chart in Overview dashboard.
The filters panel displays additional filters that are relevant to external applications tracing data (for example, Business ID filter).
Architecture
Within the external application pod, alongside the custom user application, two sidecar containers exist:
Operations Dashboard Agent - Receives spans (over UDP) from the custom user application, decides which spans are to be sampled, and forwards those to the Operations Dashboard Collector (over TCP).
Operations Dashboard Collector - Receives spans (over TCP) from the Operations Dashboard Agent and forwards them to the Operations Dashboard Store (over TLS).
Operations Dashboard registration service is an API for registering the external application to Operations Dashboard.
A secret allows Operations Dashboard Collector to securely communicate with the Operations Dashboard Store, and is created during the process of registering the external application to Operations Dashboard.
Integrating Operations Dashboard components into the external application workload
Obtaining images
Operations Dashboard Agent and Collector images are available in IBM Entitled Registry. To use the IBM Entitled Registry, you must first obtain an Entitlement key, which can be obtained from https://myibm.ibm.com/products-services/containerlibrary (IBM Container Library).
The version of the images should be the same as the version of the Operations Dashboard instance that is targeted to receive the tracing data. Please change <OD_NAMESPACE> in the command below to the namespace where Operations Dashboard is installed in your environment and execute this command to inspect the version of the images:
oc get statefulset -n <OD_NAMESPACE> -o jsonpath='{range .items[*].spec.template.spec.containers[*]}{.image}{"\n"}{end}'In order to pull the images from IBM Entitled Registry, change the following values in the commands below and execute them:
<YOUR_ENTITLED_REGISTRY_KEY>- the Entitlement key obtained above.<IMAGES_VERSION>- The version of the images inspected above (appears twice).
docker login cp.icr.io --username cp --password <YOUR_ENTITLED_REGISTRY_KEY>
docker pull cp.icr.io/cp/icp4i/ace/icp4i-od-agent:<IMAGES_VERSION>-amd64
docker pull cp.icr.io/cp/icp4i/ace/icp4i-od-collector:<IMAGES_VERSION>-amd64Adding sidecar containers
Operations Dashboard Agent and Collector should be added to the external application workload (statefulset / deployment) as sidecar containers.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: icp4i-od-external-app-generator
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: icp4i-od-external-app-app
spec:
selector:
matchLabels:
helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: icp4i-od-external-app-app
serviceName: icp4i-od-external-app-generator
replicas: 1
template:
metadata:
annotations:
labels:
helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: icp4i-od-external-app-app
spec:
affinity:
{{- if .Values.image.pullSecret }}
imagePullSecrets:
- name: {{ .Values.image.pullSecret }}
{{- end }}
volumes:
- name: icp4i-od-ca-certificate
secret:
secretName: icp4i-od-store-cred
optional: true
containers:
####################################################
############### External App ###############
####################################################
- name: od-external-app
image: "{{ trimSuffix "/" .Values.image.repository }}/{{ .Values.image.odExternalApp }}"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 256m
memory: 128Mi
readinessProbe:
exec:
command:
- /bin/sh
- -c
- /usr/local/bin/check_process.sh
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
livenessProbe:
exec:
command:
- /bin/sh
- -c
- /usr/local/bin/check_process.sh
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
##########################################################
############### Sidecar - OD Agent ###############
##########################################################
- name: od-tracing-agent
# example : image-registry.openshift-image-registry.svc:5000/myNamespace/icp4i-od-agent:1.0.2-amd64
image: "{{ trimSuffix "/" .Values.image.repository }}/{{ .Values.image.odAgent }}"
command: ["/go/bin/agent-linux"]
args:
- "--agent.tags=sourceNamespace={{ .Release.Namespace }}"
ports:
- containerPort: 5775
protocol: UDP
- containerPort: 6831
protocol: UDP
- containerPort: 6832
protocol: UDP
- containerPort: 5778
protocol: TCP
env:
- name: COLLECTOR_HOST_PORT
value : localhost:14267
- name: REPORTER_TYPE
value : "grpc"
- name: REPORTER_GRPC_HOST_PORT
value : localhost:14250
- name: REPORTER_GRPC_TLS
value : "false"
- name: OD_SOURCE_NAMESPACE
value : {{ .Release.Namespace }}
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 256m
memory: 128Mi
readinessProbe:
httpGet:
path: /?service=health
port: 5778
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
livenessProbe:
tcpSocket:
port: 5778
initialDelaySeconds: 8
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
##############################################################
############### Sidecar - OD Collector ###############
##############################################################
- name: od-tracing-collector
# example : image-registry.openshift-image-registry.svc:5000/myNamespace/icp4i-od-collector:1.0.2-amd64
image: "{{ trimSuffix "/" .Values.image.repository }}/{{ .Values.image.odCollector }}"
volumeMounts:
- name: icp4i-od-ca-certificate
mountPath: "/usr/share/jaeger/ca/certs"
ports:
- containerPort: 14250
name: grpc
protocol: TCP
- containerPort: 14267
name: tchannel
protocol: TCP
- containerPort: 14268
name: http
protocol: TCP
- containerPort: 14269
name: healthcheck
protocol: TCP
- containerPort: 9411
name: zipkin
protocol: TCP
- containerPort: 14987
name: liveliness
protocol: TCP
env:
- name: SPAN_STORAGE_TYPE
value: elasticsearch
- name: ES_PASSWORD
valueFrom:
secretKeyRef:
name: icp4i-od-store-cred
key: password
- name: ES_USERNAME
valueFrom:
secretKeyRef:
name: icp4i-od-store-cred
key: username
- name: ES_SERVER_URLS
value: https://od-store-od.icp4i-od-ns.svc:9200
- name: ES_TLS_CA
value: '/usr/share/jaeger/ca/certs/icp4i-od-cacert.pem'
- name: COLLECTOR_PORT
value: "14267"
- name: COLLECTOR_HTTP_PORT
value: "14268"
- name: COLLECTOR_ZIPKIN_HTTP_PORT
value: "9411"
- name: COLLECTOR_GRPC_TLS
value: "false"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 256m
memory: 128Mi
readinessProbe:
httpGet:
port: 14269
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
livenessProbe:
tcpSocket:
port: 14987
initialDelaySeconds: 8
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3Changes to the external application code
In order to generate tracing data within the application, the following needs to be added to the application code:
Create spans using OpenTracing API or one of OpenTracing API contributions that include the following tags:
sourceNamespace (mandatory): The namespace where the external application is deployed. This should be set as a process tag by the agent (see
agent.tagsparameter in the example above).service (mandatory): The name of the external application determined by the customer. This should be set as the service name for Jaeger Tracer (see
JAEGER_SERVICE_NAMEenvironment variable in the example below). For example: "digital-banking-apis", "CRM", etc.externalAppType (mandatory): application classification determined by the customer and displayed in the Web Console. For example: "mobile-app", "core-app", etc.
operation (mandatory): The operation name determined by the customer. For example: "make-deposit", "create-account", etc.
span.kind (optional): Technical classification of the operation determined by the customer. For example: "invoke-rest-api", "kafka-producer", etc.
peer.service (optional): The name of the invoked service (application)in cases such as a REST API invocation or message delivery.
message_bus.destination (optional): In case of message delivery, the name of the queue or topic a message is delivered to or consumed from.
partition (optional): Kafka partition if applicable.
offset (optional): Kafka message offset if applicable.
businessId (optional): a business correlation ID that may be used in the Web Console filters.
Send the spans to the Operations Dashboard Agent using a Tracer implemented by one of Jaeger client libraries.
Code example
Consider a Java application that connects to Event Streams for producing and consuming messages. The following example demonstrates the changes such an application needs to perform in order to generate tracing data, assuming the application uses Maven for dependency management.
pom.xml
Add the following dependencies:
<dependency>
<groupId>io.opentracing.contrib</groupId>
<artifactId>opentracing-kafka-client</artifactId>
<version>0.1.11</version>
</dependency>
<dependency>
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-client</artifactId>
<version>1.1.0</version>
</dependency>opentracing-kafka-client is an OpenTracing instrumentation for Apache Kafka client that provides decorated Producer and Consumer classes for automatically generating spans when messages are produced or consumed.
Creating a Jaeger Tracer
Usually a Tracer is configured based on environment variables. For example:
JAEGER_AGENT_HOST: Should be
127.0.0.1since the Agent is running within the same pod of the external application as a sidecar container.JAEGER_AGENT_PORT: Usually
6831JAEGER_SERVICE_NAME: The name of the external application. See service tag description above.
JAEGER_SAMPLER_TYPE: Jaeger Sampler type. For example:
constJAEGER_SAMPLER_PARAM: Jaeger sampler param. For example:
1
To create a Tracer object using environment variables, make sure to initialize the environment variables and use the following code:
Tracer tracer = Configuration.fromEnv().getTracer();Generating spans
Once the Tracer is available, when producing or consuming Kafka messages, use TracingKafkaProducer and TracingKafkaConsumer instead of the standard KafkaProducer and KafkaConsumer. For example:
TracingKafkaProducerBuilder<String, String> builder = new TracingKafkaProducerBuilder<String, String>(kafkaProducer, tracer);
builder.withDecorators(Arrays.asList(SpanDecorator.STANDARD_TAGS, new TracingSpanDecorator(externalAppType, businessId)));
try (TracingKafkaProducer<String, String> tracingKafkaProducer = builder.build()) {
tracingKafkaProducer.send(new ProducerRecord<>(kafkaTopicName, recordContent));
}kafkaProduceris the standard object for producing messages. It is assumed that the external application already has such an object to decorate.traceris the tracer object created before.TracingSpanDecoratoris a new class that needs to be created to set span tags - see below for more information.externalAppType- the content of externalAppType tag as discussed above.businessId- the content of businessId tag as discussed above.kafkaTopicName- the topic name to produce a message to. It is assumed that the external application already has this.recordContent- the content of the produced messaeg. It is assumed that the external application already has this.
Adding required span tags
opentracing-kafka-client allows adding custom tags to spans using a SpanDecorator class. See the following example for a SpanDecorator that adds the externalAppType and businessId tags:
public static class TracingSpanDecorator implements SpanDecorator {
private static final String EXTERNAL_APP_TYPE_TAG = "externalAppType";
private static final String BUSINESS_ID_TAG = "businessId";
private final String externalAppTypeTag; // Mandatory tag!
private final String businessIdTag;
public TracingSpanDecorator(String externalAppTypeTag, String businessIdTag) {
this.externalAppTypeTag = externalAppTypeTag;
this.businessIdTag = businessIdTag;
}
@Override
public <K, V> void onSend(ProducerRecord<K, V> record, Span span) {
span.setTag(EXTERNAL_APP_TYPE_TAG, externalAppTypeTag);
span.setTag(BUSINESS_ID_TAG, businessIdTag);
}
@Override
public <K, V> void onResponse(ConsumerRecord<K, V> record, Span span) {
span.setTag(EXTERNAL_APP_TYPE_TAG, externalAppTypeTag);
span.setTag(BUSINESS_ID_TAG, businessIdTag);
}
@Override
public void onError(Exception exception, Span span) {
// Do nothing
}
}Registering the external application to Operations Dashboard
After the external application is installed, it should register to Operations Dashboard so it can send the tracing data. See Capability registration for more information.
Capability registration API (recommended)
IBM recommends that you use the capability registration API, if you can meet the prerequisites.
Simply create the secret and the custom resource as documented. The Agent and Collector containers start, and the external application instance is ready.
Manual capability registration
The manual registration is usually implemented as a job that continuously invokes registration requests until it succeeds (see details in the manual command described below). To manually send a registration request to Operations Dashboard from one of the external application's containers, change the following values in the command below and execute it:
<EXTERNAL_APPLICATION_POD_NAME>- The external application pod name.<EXTERNAL_APPLICATION_CONTAINER_NAME>- The external application container name.<OD_NAMESPACE>- The namespace where Operations Dashboard is deployed.<EXTERNAL_APPLICATION_NAMESPACE>- The namespace where the external application is deployed.
oc exec -it <EXTERNAL_APPLICATION_POD_NAME> -c <EXTERNAL_APPLICATION_CONTAINER_NAME> -- curl -s -X GET --insecure https://icp4i-od.<OD_NAMESPACE>.svc:8090/tracing/register/<EXTERNAL_APPLICATION_NAMESPACE>/<External Application POD name> >/dev/nullComplete the registration process:
A new registration request appears on Registration requests page. If it doesn't, have a look at Troubleshooting page.
As long as the registration process is not complete, the Agent and Collector sidecar containers will not become ready.
Once the registration request is approved, the Operations Dashboard administrator is displayed with a command they need to execute in order to create a secret in the namespace of the external application.
Once the command is executed, and the secret is created, the process is complete and the external application instance should become ready.