Configuring an API instance for AI test generation

AI test generation analyzes and compares OpenTracing traces emitted by your application, so it is a prerequisite that your API be instrumented to produce those traces. Specifically, the analysis assumes the Jaeger implementation of OpenTracing and your application must send its trace spans to a Jaeger collector.

OpenTracing is an aspect of OpenTelemetry and, if you don't yet have tracing enabled in your application, consider using the OpenTelemetry libraries as a quick way to get started. The analysis primarily examines calls between RESTful services and does not yet fully take into account spans generated within a service, so a minimal trace that records only incoming and outgoing calls is sufficient.

Your API must be a RESTful API that is described by an OpenAPI 2.0 (Swagger 2.0) definition.

AI Test Generation operates across two distinct environments:

• Production environment, where traces are produced.

• Test environment, where the traces received from the production environment are analyzed, test cases are generated, and tests are run.

For the sake of illustration, this procedure assumes that each environment corresponds to a single OpenShift project, and that all the components required for an environment are deployed within the corresponding project. The two projects may be in different clusters, but a requirement of this release is that the Data service deployed to the Production environment must be directly reachable from the Test environment.

Prerequisites: Both environments require that you install IBM Cloud Pak® for Integration as described in Overview – Installation. In particular:

• Install the IBM API Connect operator, using the procedure in Installing the operators by using the CLI or Installing the operators by using the Red Hat OpenShift console.

• Create a Cloud Pak for Integration instance, using the procedure in Deploying the Platform UI by using the CLI or Deploying the Platform UI by using the OpenShift console.

AI Test Generation is compatible with the Red Hat OpenShift distributed tracing platform (previously Red Hat OpenShift Jaeger) using the production deployment strategy and supported by Elasticsearch storage. Follow the instructions for Installing Jaeger in the OpenShift Container Platform documentation. You can install Jaeger with or without Service Mesh, depending on your requirements. To improve security for the Jaeger Query gRPC interface, follow the steps in Securing Jaeger Query with TLS or mTLS.

As an example, assume that your Jaeger instance is called jaeger-production. Once installation is complete, you should find the following resources:

• A service called jaeger-production-collector-headless.

• A secret called jaeger-production-collector-headless-tls.

You will need these resources when configuring your API management instance later in this procedure.

Creating the Jaeger Query service

Create a service called jaeger-production-query-api that exposes the trace data for analysis. You can create the service in the Red Hat OpenShift console or the CLI.

To create the service in the Red Hat OpenShift console:

1. Click Networking > Services.

2. Select the project that contains the Jaeger instance.

3. Click Create Service.

4. Select the YAML view.

5. Replace the sample YAML with the following service instance. Use the actual name of your instance for the value of jaeger-production:

   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger-production-query-api
     labels:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query-api
   spec:
     selector:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query
     ports:
       - name: grpc-api
         protocol: TCP
         port: 16685
       - name: http-api
         protocol: TCP
         port: 16686

6. Click Create.

To create the service from the CLI:

1. Copy the following service definition into a text editor. Use the actual name of your instance for the value of jaeger-production:

   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger-production-query-api
     labels:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query-api
   spec:
     selector:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query
     ports:
       - name: grpc-api
         protocol: TCP
         port: 16685
       - name: http-api
         protocol: TCP
         port: 16686

2. Save the text to a file called jaeger-query-api-service.yaml.

3. Run this command; use your actual project name as the value for {my-project}:

   oc apply -f jaeger-query-api-service.yaml -n {my-project}

Creating a Jaeger Query network policy

Important: Because the Jaeger Query service exposes an unsecured port throughout your cluster, IBM recommends you also create a network policy that limits access to the service to only the AI Test Generation Data service deployed in the same project.

You can create the network policy in the Red Hat OpenShift console or the CLI.

To create the network policy in the Red Hat OpenShift console:

1. Click Networking > Network Policies.

2. Select the project that contains the Jaeger instance.

3. Click Create Network Policy.

4. Select the YAML view.

5. Replace the sample YAML with the following policy definition. Replace jaeger-production with actual name of your instance:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: jaeger-production-query
   spec:
     podSelector:
       matchLabels:
         app: jaeger
         app.kubernetes.io/component: query
         app.kubernetes.io/name: jaeger-production-query
     ingress:
       - ports:
           - protocol: TCP
             port: 16685
         from:
           - podSelector:
               matchLabels:
                 app.kubernetes.io/managed-by: ibm-apiconnect
                 app.kubernetes.io/name: api-testgen-data
       - ports:
           - protocol: TCP
             port: 8443
         from:
           - namespaceSelector: {}
     policyTypes:
       - Ingress

6. Click Create.

To create the network policy in the CLI:

1. Copy the following policy definition into a text editor. Replace jaeger-production with actual name of your instance:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: jaeger-production-query
   spec:
     podSelector:
       matchLabels:
         app: jaeger
         app.kubernetes.io/component: query
         app.kubernetes.io/name: jaeger-production-query
     ingress:
       - ports:
           - protocol: TCP
             port: 16685
         from:
           - podSelector:
               matchLabels:
                 app.kubernetes.io/managed-by: ibm-apiconnect
                 app.kubernetes.io/name: api-testgen-data
       - ports:
           - protocol: TCP
             port: 8443
         from:
           - namespaceSelector: {}
     policyTypes:
       - Ingress

2. Save the text to a file called jaeger-query-api-policy.yaml.

3. Run this command; replace {my-project} with the name of your project that contains the Jaeger instance:

   oc apply -f jaeger-query-api-policy.yaml -n {my-project}

Although how you deploy your API is beyond the scope of this document, your API must be configured to log trace spans to the Jaeger collector you created in the previous section.

At this time, the analysis by API Analytics only works for simple RESTful services implemented directly using programming languages such as JavaScript, Go, Python,, and so on, and there are OpenTelemetry libraries to assist with instrumentation for these languages. For more information, see the OpenTelemetry documentation. Cloud Pak for Integration capabilities such as IBM MQ and IBM App Connect do not yet log trace spans to Jaeger, so they are excluded from the analysis.

Prerequisite step: If you do not have a deployed instance of API management, follow the procedure in API management deployment to create a minimal deployment.

1. Log in to the IBM Cloud Pak Platform UI.

2. In the navigation menu, click Administration > Integration instances.

3. Select your instance.

4. Click the overflow menu (three-dot icon) and click Edit.

5. Set Advanced settings to on so that you can enable advanced functionality for the instance that supports AI Test Generation.

6. Configure the API gateway so that it logs trace spans to Jaeger.

   1. Select Gateway subsystem.

   2. Under Jaeger Tracing, scroll down and set the Jaeger collector endpoint to be the collector endpoint of the Jaeger instance you created in Installing Jaeger in your Production environment. In the example, the Jaeger agent deployed with the API Connect gateway communicates with the collector service on port 14250 (HTTPS gRPC), so the appropriate endpoint based on the example instance name jaeger-production would be:

      jaeger-production-collector-headless:14250

      • Set the TLS Secret name as the secret you identified in Installing Jaeger in your Production environment.

      • Set the TLS Server name as the full cluster local DNS name for the collector service. Based on the example instance name jaeger-production in the project my-project the server name would be:

        jaeger-production-collector-headless.my-project.svc.cluster.local

7. Click Create to deploy your settings.

For more information on how to deploy and configure API management, see:

• API management deployment

• Managing your APIs

To deploy the Data service that serves trace and payload data to the analyzer:

1. Log in to IBM Cloud Pak Platform UI.

2. Follow the instructions from API management deployment through step 9.

3. Select Management subsystem

4. Enable the AI data service.

5. Set the Jaeger endpoint for API data. The endpoint you specified for the Gateway is the collector endpoint, and this is the query endpoint that provides access to the collected traces. Specify the name of the service you created earlier, on port 16685 (older versions of Jaeger, pre 1.22, use 16686):

   jaeger-production-query-api:16685

   Alternatively, you can express the configuration as a Custom Resource definition. Select the YAML view and copy and paste the following example:

   apiVersion: apiconnect.ibm.com/v1beta1
   kind: APIConnectCluster
   metadata:
     name: apiconnect-production
     labels:
       app.kubernetes.io/instance: apiconnect
       app.kubernetes.io/managed-by: ibm-apiconnect
       app.kubernetes.io/name: apiconnect-minimum
   spec:
     license:
       accept: true
       use: nonproduction
       license: <my_license>
     profile: n1xc7.m48
     version: "10.0.4"
     gateway:
       jaegerTracing:
         jaegerCollectorEndpoint: "jaeger-production-collector-headless:14250"
         tls:
           collectorEndpoint:
             secretName: jaeger-production-collector-headless-tls
             serverName: jaeger-production-collector-headless.my-project.svc.cluster.local
     management:
       aiDataService:
         enabled: true
         jaegerEndpoint: "jaeger-production-query-api:16685"

6. Click Create to deploy the API management capability, and the AI Test Generation Data service, into your project.

To enable AI insight in a test suite (at the end of this procedure), you must provide details about the AI Test generation data service deployed to your Production environment.

To get those details, run the following commands using the OpenShift CLI. In each step, replace:

• apiconnect-production with the actual name of your API management instance.

• my-project with the name of the project which contains that instance.

Data service endpoint:

oc get route -n my-project --selector app.kubernetes.io/part-of=apiconnect-production -o json | \
        jq -r '.items[] | select(.metadata.name|endswith("-api-data-service")) | ("https://" + .spec.host)'

Data service CA certificate:

oc get route -n my-project --selector app.kubernetes.io/part-of=apiconnect-production -o json | \
        jq -r '.items[] | select(.metadata.name|endswith("-api-data-service")) | .spec.tls.caCertificate'

Data service API key:

oc get secret -n my-project --selector app.kubernetes.io/part-of=apiconnect-production -o json | \
        jq -r '.items[] | select(.metadata.name|endswith("-data-service-api-key")) | .data.API_KEY' | \
        base64 --decode

To install Jaeger in the Test environment:

1. Follow the instructions for Installing Jaeger in the OpenShift Container Platform documentation. You can install Jaeger with or without Service Mesh, depending on your requirements.

   Assuming an example name of jaeger-test for your Jaeger instance, the installation should create the following additional resources, which you will need when configuring the API management instance later:

   • A service called jaeger-test-collector-headless.

   • A secret called jaeger-test-collector-headless-tls.

2. Create a service called jaeger-test-query-api that will expose the test trace data for analysis. Under Installing Jaeger in your Production environment, see "Creating the Jaeger Query service" for the procedure.

3. IBM recommends creating a network policy that limits access to only the AI Test Generation Data service deployed in the same project. Under Installing Jaeger in your Production environment, see "Creating a Jaeger network policy" for more information.

4. To improve security for the Jaeger Query gRPC interface follow these steps.

You must deploy an instance of your API into the Test environment to execute test cases (under the control of the API management capability) and it must be configured to log trace spans to the Jaeger collector you created in the previous step. How you deploy your API is beyond the scope of this document, but for general guidance refer to Deploying your API.

Prerequisite step: If you do not have a deployed instance of API management, follow the procedure in API management deployment to create a minimal deployment.

1. Log in to the IBM Cloud Pak Platform UI.

2. In the navigation menu, click Administration > Integration instances.

3. Select your instance.

4. Set Advanced settings to on. This setting enables additional functionality for the instance that supports Test and Monitor with AI insight.

5. Configure the API gateway so that it logs trace spans to Jaeger.

   1. Select Gateway subsystem.

   2. Under Jaeger Tracing, scroll to Jaeger collector endpoint and set the value as the collector endpoint of the Jaeger instance you created in Installing Jaeger in your Test environment. In the example, the Jaeger agent deployed with the API Connect gateway communicates with the collector service on port 14250 (HTTPS gRPC), so the appropriate endpoint based on the example instance name jaeger-test is:

      jaeger-test-collector-headless:14250

      • Set the TLS Secret name as the secret you identified in Installing Jaeger in your Test environment.

      • Set the TLS Server name as the full cluster local DNS name for the collector service. Based on the example instance name jaeger-test in the project my-project the server name is:

        jaeger-production-collector-headless.my-project.svc.cluster.local

6. Click Create to deploy your settings.

7. Set the TLS Secret name to be the secret you created in Installing Jaeger in your Test environment.

8. Set the TLS Server name to be the full cluster local DNS name for the collector service. Based on the example instance name jaeger-test in the project my-project the server name would be:

   jaeger-test-collector-headless.my-project.svc.cluster.local

9. Enable Test and Monitor with AI insight:

   1. Go to Management subsystem.

   2. Scroll to Test and Monitor and click Enable Test and Monitor.

   3. Click Enable AI test insight.

   4. Set the Jaeger endpoint for API data. The endpoint you specified for the gateway is the collector endpoint, and this endpoint is also the query endpoint that provides access to the collected traces. Use one of these two options for setting the endpoint:

      • Specify the name of the service you created earlier, on port 16685. For an older versions of Jaeger (before version 1.22), use 16686. For example:

        jaeger-test-query-api:16685

      • Express the configuration as a custom resource (CR) definition. Click the YAML view and copy and paste the following example:

        apiVersion: apiconnect.ibm.com/v1beta1
        kind: APIConnectCluster
        metadata:
          name: apiconnect-test
          labels:
            app.kubernetes.io/instance: apiconnect
            app.kubernetes.io/managed-by: ibm-apiconnect
            app.kubernetes.io/name: apiconnect-minimum
        spec:
          license:
            accept: true
            use: nonproduction
            license: MY-LICENSE
          profile: n1xc7.m48
          version: "10.0.4"
          gateway:
            jaegerTracing:
              jaegerCollectorEndpoint: "jaeger-test-collector-headless:14250"
              tls:
                collectorEndpoint:
                  secretName: jaeger-test-collector-headless-tls
                  serverName: jaeger-test-collector-headless.my-project.svc.cluster.local
          management:
            testAndMonitor:
              enabled: true
              aiEnabled: true
              jaegerEndpoint: "jaeger-test-query-api.cp4i.svc:16685"

10. Click Create to deploy the API management capability, with Test and Monitor and AI insight, into your project.

For more information on how to deploy and configure API management, see:

• API management deployment

• Managing your APIs

In the API Manager UI:

1. Select Develop APIs and products.

2. Open your API.

3. Select the Test tab.

4. Send some test requests.

If you see the correct responses, this confirms that your API is operating through the gateway.

In the Red Hat OpenShift console:

1. Click Networking then Routes.

2. Select the project for your environment.

3. Search for your Jaeger instance (jaeger-production or jaeger-test).

4. Click the Location link to open the Jaeger UI.

5. Set the Service to apiconnect.

6. Click Find Traces.

If you see traces corresponding to your test API calls, this confirms that the gateway and your API are logging trace spans to Jaeger.