External applications tracing data

Applications without an integrated IBM Cloud Pak for Integration tracing support (e.g. a Java application that connects to Event Streams for producing and consuming messages) can also push tracing data into the Operations Dashboard runtime. Such applications are called external applications in the Web Console.

Overview

In order to send and view tracing data from external applications, the following requirements must be met:

  • Operations Dashboard Web Console needs to be configured to display external applications tracing data. See display system parameters for more information.

  • The external application needs to be deployed inside the IBM Cloud Pak for Integration cluster.

  • Operations Dashboard components must be integrated into the external application workload (see below).

  • Code changes needs to be made to the external application code (see below).

  • Registration process of the exernal application must be completed (see below).

Once the requirements are met, tracing data will be sent to Operations Dashboard and will be displayed in the Web Console:

  • Ext. app overview dashboard will be displayed.

  • External applications data will be displayed as an additional capability in "New traces by capability" chart in overview dashboard.

  • The filters panel will show additional filters relevant for external applications tracing data (e.g. Business ID filter).

Architecture

Architecture
  • Within the external application pod, alongside the custom user application, two sidecar containers exist:

    • Operations Dashboard Agent - Receives spans (over UDP) from the custom user application, decides which spans are to be sampled, and forwards those to the Operations Dashboard Collector (over TCP).

    • Operations Dashboard Collector - Receives spans (over TCP) from the Operations Dashboard Agent and forwards them to the Operations Dashboard Store (over TLS).

  • Operations Dashboard registration service is an API for registering the external application to Operations Dashboard.

  • A secret allows Operations Dashboard Collector to securely communicate with the Operations Dashboard Store, and is created during the process of registering the external application to Operations Dashboard.

Integrating Operations Dashboard components into the external application workload

Obtaining images

Operations Dashboard Agent and Collector images are available in IBM Entitled Registry. To use the IBM Entitled Registry, you must first obtain an Entitlement key, which can be obtained from https://myibm.ibm.com/products-services/containerlibrary (IBM Container Library).

The version of the images should be the same as the version of the Operations Dashboard instance that should receive the tracing data. Please change <OD_NAMESPACE> in the command below to the namesapce where Operations Dashboard is installed in your environment and execute the following command to inspect the version of the images:

oc get statefulset -n <OD_NAMESPACE> -o jsonpath='{range .items[*].spec.template.spec.containers[*]}{.image}{"\n"}{end}'

In order to pull the images from IBM Entitled Registry, please change the following values in the commands below and execute them:

  • <YOUR_ENTITLED_REGISTRY_KEY> - the Entitlement key obtained above.

  • <IMAGES_VERSION> - The version of the images inspected above (appears twice).

docker login cp.icr.io --username cp --password <YOUR_ENTITLED_REGISTRY_KEY>
docker pull cp.icr.io/cp/icp4i/ace/icp4i-od-agent:<IMAGES_VERSION>-amd64
docker pull cp.icr.io/cp/icp4i/ace/icp4i-od-collector:<IMAGES_VERSION>-amd64

Adding sidecar containers

Operations Dashboard Agent and Collector should be added to the external application workload (statefulset / deployment) as sidecar containers.

For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: icp4i-od-external-app-generator
  labels:
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: icp4i-od-external-app-app
spec:
  selector:
    matchLabels:
      helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
      app.kubernetes.io/instance: {{ .Release.Name }}
      app.kubernetes.io/component: icp4i-od-external-app-app
  serviceName: icp4i-od-external-app-generator
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
        app.kubernetes.io/instance: {{ .Release.Name }}
        app.kubernetes.io/component: icp4i-od-external-app-app
    spec:
      affinity:
      {{- if .Values.image.pullSecret }}
      imagePullSecrets:
        - name: {{ .Values.image.pullSecret }}
      {{- end }}
      volumes:
      - name: icp4i-od-ca-certificate
        secret:
          secretName: icp4i-od-store-cred
          optional: true
      containers:

      ####################################################
      ###############     External App     ###############
      ####################################################

      - name: od-external-app
        image: "{{ trimSuffix "/" .Values.image.repository }}/{{ .Values.image.odExternalApp }}"
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 256m
            memory: 128Mi
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - /usr/local/bin/check_process.sh
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3  
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - /usr/local/bin/check_process.sh
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3  

      ##########################################################
      ###############     Sidecar - OD Agent     ###############
      ##########################################################
 
      - name: od-tracing-agent
        # example :  image-registry.openshift-image-registry.svc:5000/myNamespace/icp4i-od-agent:1.0.2-amd64
        image: "{{ trimSuffix "/" .Values.image.repository }}/{{ .Values.image.odAgent }}"
        command: ["/go/bin/agent-linux"]
        args: 
          - "--jaeger.tags=sourceNamespace={{ .Release.Namespace }}"
        ports:
          - containerPort: 5775
            protocol: UDP
          - containerPort: 6831
            protocol: UDP
          - containerPort: 6832
            protocol: UDP
          - containerPort: 5778
            protocol: TCP            
        env:
          - name: COLLECTOR_HOST_PORT
            value : localhost:14267
          - name: REPORTER_TYPE
            value : "grpc"
          - name: REPORTER_GRPC_HOST_PORT
            value : localhost:14250              
          - name: REPORTER_GRPC_TLS
            value : "false"
          - name: OD_SOURCE_NAMESPACE  
            value : {{ .Release.Namespace }}
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 256m
            memory: 128Mi
        readinessProbe:
          httpGet:
            path: /?service=health
            port: 5778
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3  
        livenessProbe:
          tcpSocket:
            port: 5778
          initialDelaySeconds: 8
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
      
      ##############################################################
      ###############     Sidecar - OD Collector     ###############
      ##############################################################

      - name: od-tracing-collector
        # example : image-registry.openshift-image-registry.svc:5000/myNamespace/icp4i-od-collector:1.0.2-amd64
        image: "{{ trimSuffix "/" .Values.image.repository }}/{{ .Values.image.odCollector }}"
        volumeMounts:
          - name: icp4i-od-ca-certificate
            mountPath: "/usr/share/jaeger/ca/certs" 
        ports:            
          - containerPort: 14250
            name: grpc
            protocol: TCP
          - containerPort: 14267
            name: tchannel
            protocol: TCP
          - containerPort: 14268
            name: http
            protocol: TCP
          - containerPort: 14269
            name: healthcheck
            protocol: TCP
          - containerPort: 9411
            name: zipkin
            protocol: TCP
          - containerPort: 14987
            name: liveliness
            protocol: TCP
        env:            
          - name: SPAN_STORAGE_TYPE
            value: elasticsearch            
          - name: ES_PASSWORD
            valueFrom:
              secretKeyRef:
                name: icp4i-od-store-cred
                key: password
          - name: ES_USERNAME
            valueFrom:
              secretKeyRef:
                name: icp4i-od-store-cred
                key: username
          - name: ES_SERVER_URLS
            value: https://od-store-od.icp4i-od-ns.svc:9200 
          - name: ES_TLS_CA     
            value: '/usr/share/jaeger/ca/certs/icp4i-od-cacert.pem'          
          - name: COLLECTOR_PORT
            value: "14267"
          - name: COLLECTOR_HTTP_PORT
            value: "14268"
          - name: COLLECTOR_ZIPKIN_HTTP_PORT
            value: "9411"
          - name: COLLECTOR_GRPC_TLS
            value: "false"  
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 256m
            memory: 128Mi              
        readinessProbe:
          httpGet:
            port: 14269
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3  
        livenessProbe:
          tcpSocket:
            port: 14987
          initialDelaySeconds: 8
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3

Changes to the external application code

In order to generate tracing data within the application, the following needs to be added to the application code:

  • Create spans using OpenTracing API or one of OpenTracing API contributions that include the following tags:

    • sourceNamespace (mandatory): The namespace where the external application is deployed. This should be set as a process tag by the agent (see jaeger.tags parameter in the example above).

    • service (mandatory): The name of the external application determined by the customer. This should be set as the service name for Jaeger Tracer (see JAEGER_SERVICE_NAME environment variable in the example below). For example: "digital-banking-apis", "CRM", etc.

    • externalAppType (mandatory): application classification determined by the customer and displayed in the Web Console. For example: "mobile-app", "core-app", etc.

    • operation (mandatory): The operation name determined by the customer. For example: "make-deposit", "create-account", etc.

    • span.kind (optional): Technical classification of the operation determined by the customer. For example: "invoke-rest-api", "kafka-producer", etc.

    • peer.service (optional): The name of the invoked service (application)in cases such as a REST API invocation or message delivery.

    • message_bus.destination (optional): In case of message delivery, the name of the queue or topic a message is delivered to or consumed from.

    • partition (optional): Kafka partition if applicable.

    • offset (optional): Kafka message offset if applicable.

    • businessId (optional): a business correlation ID that may be used in the Web Console filters.

  • Send the spans to the Operations Dashboard Agent using a Tracer implemented by one of Jaeger client libraries.

Code example

Consider a Java application that connects to Event Streams for producing and consuming messages. The following example demonstrates the changes such an application needs to perform in order to generate tracing data, assuming the application uses Maven for dependency management.

pom.xml

Add the following dependencies:

<dependency>
	<groupId>io.opentracing.contrib</groupId>
	<artifactId>opentracing-kafka-client</artifactId>
	<version>0.1.11</version>
</dependency>

<dependency>
	<groupId>io.jaegertracing</groupId>
	<artifactId>jaeger-client</artifactId>
	<version>1.1.0</version>
</dependency>

opentracing-kafka-client is an OpenTracing instrumentation for Apache Kafka client that provides decorated Producer and Consumer classes for automatically generating spans when messages are produced or consumed.

Creating a Jaeger Tracer

Usually a Tracer is configured based on environment variables. For example:

  • JAEGER_AGENT_HOST: Should be 127.0.0.1 since the Agent is running within the same pod of the external application as a sidecar container.

  • JAEGER_AGENT_PORT: Usually 6831

  • JAEGER_SERVICE_NAME: The name of the external application. See service tag description above.

  • JAEGER_SAMPLER_TYPE: Jaeger Sampler type. For example: const

  • JAEGER_SAMPLER_PARAM: Jaeger sampler param. For example: 1

To create a Tracer object using environment variables, make sure to initialize the environment variables and use the following code:

Tracer tracer = Configuration.fromEnv().getTracer();

Generating spans

Once the Tracer is available, when producing or consuming Kafka messages, use TracingKafkaProducer and TracingKafkaConsumer instead of the standard KafkaProducer and KafkaConsumer. For example:

TracingKafkaProducerBuilder<String, String> builder = new TracingKafkaProducerBuilder<String, String>(kafkaProducer, tracer);
builder.withDecorators(Arrays.asList(SpanDecorator.STANDARD_TAGS, new TracingSpanDecorator(externalAppType, businessId)));
try (TracingKafkaProducer<String, String> tracingKafkaProducer = builder.build()) {
	tracingKafkaProducer.send(new ProducerRecord<>(kafkaTopicName, recordContent));
}
  • kafkaProducer is the standard object for producing messages. It is assumed that the external application already has such an object to decorate.

  • tracer is the tracer object created before.

  • TracingSpanDecorator is a new class that needs to be created to set span tags - see below for more information.

  • externalAppType - the content of externalAppType tag as discussed above.

  • businessId - the content of businessId tag as discussed above.

  • kafkaTopicName - the topic name to produce a message to. It is assumed that the external application already has this.

  • recordContent - the content of the produced messaeg. It is assumed that the external application already has this.

Adding required span tags

opentracing-kafka-client allows adding custom tags to spans using a SpanDecorator class. See the following example for a SpanDecorator that adds the externalAppType and businessId tags:

public static class TracingSpanDecorator implements SpanDecorator {
	private static final String EXTERNAL_APP_TYPE_TAG = "externalAppType";
	private static final String BUSINESS_ID_TAG = "businessId";

	private final String externalAppTypeTag; // Mandatory tag!
	private final String businessIdTag;

	public TracingSpanDecorator(String externalAppTypeTag, String businessIdTag) {
		this.externalAppTypeTag = externalAppTypeTag;
		this.businessIdTag = businessIdTag;
	}

	@Override
	public <K, V> void onSend(ProducerRecord<K, V> record, Span span) {
		span.setTag(EXTERNAL_APP_TYPE_TAG, externalAppTypeTag);
		span.setTag(BUSINESS_ID_TAG, businessIdTag);
	}

	@Override
	public <K, V> void onResponse(ConsumerRecord<K, V> record, Span span) {
		span.setTag(EXTERNAL_APP_TYPE_TAG, externalAppTypeTag);
		span.setTag(BUSINESS_ID_TAG, businessIdTag);
	}

	@Override
	public void onError(Exception exception, Span span) {
		// Do nothing
	}
}

Registering the external application to Operations Dashboard

After the external application is installed, it should register to Operations Dashboard to be able to send the tracing data. See Capability registration for more information. This is usually implemented as a job that continuously invokes registration requests until it succeeds (see details in the manual command described below).

To manually send a registration request to Operations Dashboard from one of the external application's containers, please change the following values in the command below and execute it:

  • <EXTERNAL_APPLICATION_POD_NAME> - The external application pod name.

  • <EXTERNAL_APPLICATION_CONTAINER_NAME> - The external application container name.

  • <OD_NAMESPACE> - The namespace where Operations Dashboard is deployed.

  • <EXTERNAL_APPLICATION_NAMESPACE> - The namespace where the external application is deployed.

oc exec -it <EXTERNAL_APPLICATION_POD_NAME> -c <EXTERNAL_APPLICATION_CONTAINER_NAME> -- curl -s -X GET --insecure https://icp4i-od.<OD_NAMESPACE>.svc:8090/tracing/register/<EXTERNAL_APPLICATION_NAMESPACE>/<External Application POD name> >/dev/null

Note:
The HTTP status code is expected to be 404 always.

Complete the registration process:

  • A new registration request will appear on registration requests page. If it doesn't, have a look at Troubleshooting page.
    Note: As long as the registration process is not complete, the Agent and the Collector sidecar containers will not become ready.

  • Once the registration request is approved, the Operations Dashboard administrator is displayed with a command they need to execute in order to create a secret in the namespace of the external application.

  • Once the command is executed, and the secret is created, the process is complete and the external application instance should become ready.