Monitoring IBM API Connect (APIC) on Red Hat OpenShift

From the OpenShift CLI, create a namespace that is named instana-agent and grant it the necessary permissions.

$ oc create ns instana-agent
$ oc adm policy add-scc-to-user privileged -z instana-agent -n instana-agent

1. Open OperatorHub and switch to the namespace instana-agent.
2. Filter by Instana and select the Instana Operator (for example, version 2.0.3 or later).

When you install the Instana Operator, use the following configuration:

• Channel: stable
• Mode: All Namespaces
• Installed Namespace: openshift-operators
• Strategy: Automatic

After installation, switch the namespace to instana-agent to create the Instana agent instance.

1. Open the Instana Operator from the OperatorHub.

2. Select the Instana Agent tab.

3. Click Create Instance.

4. Switch to YAML view and enter the configuration as follows:

apiVersion: instana.io/v1
kind: InstanaAgent
metadata:
  name: instana-agent
  namespace: instana-agent
spec:
  agent:
    configuration_yaml: |
      # Optional: Add custom agent configurations here.
    endpointHost: <your-host-endpoint>
    endpointPort: '443'
    env: {}
    key: uBp4GXpZQpKrHxMXNcvInQ
  zone:
    name: GCP US  # Considered auto-generated
  cluster:
    name: <your-cluster-name>

Once created, the Instana sgent instance is visible in the instana-agent namespace. The setup process can take approximately 4-5 minutes.

After deployment, validate that the Instana agent pods are running in the instana-agent namespace.

Each node in the cluster hosts one Instana agent pod. For example, a cluster with 2 nodes show 2 pods.

To verify, run the following commands:

$ oc get pods,ds -n instana-agent

$ oc describe ds instana-agent -n instana-agent

If any pods fail to start due to permission issues, reapply the SCC policy:

$ oc adm policy add-scc-to-user privileged -z instana-agent -n instana-agent

Then, restart the DaemonSet to rebuild the pods:

$ oc rollout restart ds instana-agent -n instana-agent

Ensure that the Instana Operator is running successfully in the desired namespace before proceeding. For example:

• Instana Operator version: 2.1.25
• Namespace: <namespace name>

You can install the Instana Operator from OperatorHub or by using the OpenShift CLI.

To install the operator from OperatorHub, see Installing agent on OpenShift.

To install the operator by using the OpenShift CLI, apply the following CatalogSource script:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM Operator Catalog
  image: 'icr.io/cpopen/ibm-operator-catalog:latest'
  publisher: IBM
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 45m

To configure the Instana agent for APIC, complete the following steps:

1. Obtain the configuration YAML from Menu > Agent Deployment > OpenShift in the Instana UI.

   Example base YAML:

   apiVersion: instana.io/v1
   kind: InstanaAgent
   metadata:
     name: instana-agent
     namespace: <namespace name>
   spec:
     zone:
       name: <zone name> # Optional
     cluster:
       name: <cluster name>
     agent:
       key: <instana-agent-key>
       downloadKey: <instana-download-key>
       endpointHost: <instana-endpoint-host>
       endpointPort: "443"
       env:
         INSTANA_LOG_LEVEL: DEBUG

2. Deploy this configuration in the DEBUG mode to validate connectivity and troubleshooting logs.

If the Instana Operator or custom resource definition (CRD) becomes corrupted, reinstall it by using the following commands:

oc delete crd/agents.instana.io

Before configuring Instana, verify APIC connectivity by using the curl commands as shown in the following examples.

You might need the following details:

• APIC Management Host URL: Obtain from the APIC Operator configuration.
• Client ID and Client Secret: Obtain from the credentials.json file downloaded from your APIC Cloud Manager.

To obtain the realm, run the following validation command:

curl -k -X GET "https://<apic-host>/api/cloud/provider/identity-providers?fields=title%2Crealm"   -H "Accept: application/yaml"   -H "Accept-Language: en-US"   -H "User-Agent: Toolkit/10.0.8.0"   -H "X-Ibm-Client-Id: <client-id>"   -H "X-Ibm-Client-Secret: <client-secret>"

To obtain the bearer token, run the following validation command:

curl -k https://<apic-host>/api/token  -H "Content-Type: application/json"  -H "accept: application/json"  -d '{"username":"<username>", "password":"<password>", "realm":"<realm>", 
 "client_id":"<client-id>", "client_secret":"<client-secret>", "grant_type":"password"}'

To enable APIC monitoring, complete the following steps:

1. Update your Instana agent configuration.yaml file as follows:

   spec:
     agent:
       configuration_yaml: |-
         com.instana.plugin.ibmapiconnect:
           remote:
             - host: '<apic-host>'
               port: '443'
               username: '<username>'
               password: '<password>'
               realm: '<realm>'
               clientId: '<client-id>'
               clientSecret: '<client-secret>'
               availabilityZone: 'API-Connect'
               poll_rate: <desired-poll-rate>

2. Apply the changes by running the following command:

   oc apply -f instana-agent.yaml

To download the API Connect SSL certificate locally, run the following command:

mkdir temp
cd temp
echo | openssl s_client -connect <apic-host>:443 | openssl x509 > ~/temp/apic.cer
ls -ltr
# Should display apic.cer

To list all pods in the specified namespace, run the following command:

oc get pods -n <namespace name>

To copy the downloaded certificate to the Instana agent pod, run the following commands:

oc cp ./apic.cer <pod name>:/tmp/apic.cer -n <namespace name>

To import the certificate inside the Instana agent container, run the following command:

oc exec -it <pod name> -n <namespace name> -- /opt/instana/agent/jvm/bin/keytool   -import -alias tempcert   -keystore /opt/instana/agent/jvm/lib/security/cacerts   -storepass changeit   -file /tmp/apic.cer   -noprompt

To extract the keystore, run the following command:

oc cp <pod name>:/opt/instana/agent/jvm/lib/security/cacerts /tmp/cacerts -n <namespace name>

To delete the existing secret, run the following command:

oc delete secret ibm-apic-ca-secret -n <namespace name> --force

To create a secret by using the updated keystore, run the following command:

oc create secret generic ibm-apic-ca-secret   --from-file=cacerts=/tmp/cacerts   -n <namespace name>

To mount the secret volume, update the Instana agent configuration as follows:

spec:
  agent:
    pod:
      volumeMounts:
        - name: apic-cert-volume
          mountPath: /opt/instana/agent/jvm/lib/security/cacerts
          subPath: cacerts
      volumes:
        - name: apic-cert-volume
          secret:
            secretName: ibm-apic-ca-secret

To apply the changes, restart the Instana agent DaemonSet by running the following command:

oc rollout restart -n <namespace name> ds instana-agent

Verify that the pods restart and reach the Running state.