Implementing AI Test Generation

AI test generation analyzes and compares OpenTracing traces emitted by your application, so it is a prerequisite that your API be instrumented to produce those traces. Specifically, the analysis assumes the Jaeger implementation of OpenTracing and your application must send its trace spans to a Jaeger collector.

OpenTracing is an aspect of OpenTelemetry and, if you don't yet have tracing enabled in your application, consider using the OpenTelemetry libraries as a quick way to get started. The analysis primarily examines calls between RESTful services and does not yet fully take into account spans generated within a service, so a minimal trace that records only incoming and outgoing calls is sufficient.

Your API must be a RESTful API that is described by an OpenAPI 2.0 (Swagger 2.0) definition.

AI Test Generation operates across two distinct environments:

• Production environment, where traces are produced.

• Test environment, where the traces received from the production environment are analyzed, test cases are generated, and tests are run.

For the sake of illustration, this procedure assumes that each environment corresponds to a single OpenShift project, and that all the components required for an environment are deployed within the corresponding project. The two projects may be in different clusters, but a requirement of this release is that the Data service deployed to the Production environment must be directly reachable from the Test environment.

Prerequisites: Both environments require that you install IBM Cloud Pak® for Integration as described in Overview – Installation. In particular:

• Install the IBM API Connect operator, using the procedure in Installing the operators using the CLI or Installing the operators using the Openshift console.

• Create a Cloud Pak for Integration instance, using the procedure in Deploying IBM Cloud Pak for Integration using the CLI or Deploying Cloud Pak for Integration using the OpenShift console.

AI Test Generation is compatible with the Red Hat OpenShift distributed tracing platform (previously Red Hat OpenShift Jaeger) using the production deployment strategy and supported by Elasticsearch storage. Follow the instructions for Installing Jaeger in the OpenShift Container Platform documentation. You can install Jaeger with or without Service Mesh, depending on your requirements. To improve security for the Jaeger Query gRPC interface, follow the steps in Securing Jaeger Query with TLS or mTLS.

As an example, assume that your Jaeger instance is called jaeger-production. Once installation is complete, you should find the following resources:

• A service called jaeger-production-collector-headless.

• A secret called jaeger-production-collector-headless-tls.

You will need these resources when configuring your API management instance later in this procedure.

Creating the Jaeger Query service

Create a service called jaeger-production-query-api that exposes the trace data for analysis. You can create the service in the Red Hat OpenShift console or the CLI.

To create the service in the Red Hat OpenShift console:

1. Click Networking > Services.

2. Select the project that contains the Jaeger instance.

3. Click Create Service.

4. Select the YAML view.

5. Replace the sample YAML with the following service instance. Use the actual name of your instance for the value of jaeger-production:

   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger-production-query-api
     labels:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query-api
   spec:
     selector:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query
     ports:
       - name: grpc-api
         protocol: TCP
         port: 16685
       - name: http-api
         protocol: TCP
         port: 16686

6. Click Create

To create the service from the CLI:

1. Copy the following service definition into a text editor. Use the actual name of your instance for the value of jaeger-production:

   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger-production-query-api
     labels:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query-api
   spec:
     selector:
       app: jaeger
       app.kubernetes.io/component: query
       app.kubernetes.io/name: jaeger-production-query
     ports:
       - name: grpc-api
         protocol: TCP
         port: 16685
       - name: http-api
         protocol: TCP
         port: 16686

2. Save the text to a file called jaeger-query-api-service.yaml.

3. Run this command; use your actual project name as the value for {my-project}:

   oc apply -f jaeger-query-api-service.yaml -n {my-project}

Creating a Jaeger Query network policy

Important: Because the Jaeger Query service exposes an unsecured port throughout your cluster, IBM recommends you also create a network policy that limits access to the service to only the AI Test Generation Data service deployed in the same project.

You can create the network policy in the Red Hat OpenShift console or the CLI.

To create the network policy in the Red Hat OpenShift console:

1. Click Networking > Network Policies.

2. Select the project that contains the Jaeger instance.

3. Click Create Network Policy.

4. Select the YAML view.

5. Replace the sample YAML with the following policy definition. Replace jaeger-production with actual name of your instance:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: jaeger-production-query
   spec:
     podSelector:
       matchLabels:
         app: jaeger
         app.kubernetes.io/component: query
         app.kubernetes.io/name: jaeger-production-query
     ingress:
       - ports:
           - protocol: TCP
             port: 16685
         from:
           - podSelector:
               matchLabels:
                 app.kubernetes.io/managed-by: ibm-apiconnect
                 app.kubernetes.io/name: api-testgen-data
       - ports:
           - protocol: TCP
             port: 8443
         from:
           - namespaceSelector: {}
     policyTypes:
       - Ingress

6. Click Create

To create the network policy in the CLI:

1. Copy the following policy definition into a text editor. Replace jaeger-production with actual name of your instance:

   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: jaeger-production-query
   spec:
     podSelector:
       matchLabels:
         app: jaeger
         app.kubernetes.io/component: query
         app.kubernetes.io/name: jaeger-production-query
     ingress:
       - ports:
           - protocol: TCP
             port: 16685
         from:
           - podSelector:
               matchLabels:
                 app.kubernetes.io/managed-by: ibm-apiconnect
                 app.kubernetes.io/name: api-testgen-data
       - ports:
           - protocol: TCP
             port: 8443
         from:
           - namespaceSelector: {}
     policyTypes:
       - Ingress

2. Save the text to a file called jaeger-query-api-policy.yaml.

3. Run this command; replace {my-project} with the name of your project that contains the Jaeger instance:

   oc apply -f jaeger-query-api-policy.yaml -n {my-project}

Although how you deploy your API is beyond the scope of this document, your API must be configured to log trace spans to the Jaeger collector you created in the previous section.

At this time, the analysis by API Analytics only works for simple RESTful services implemented directly using programming languages such as JavaScript, Go, Python,, and so on, and there are OpenTelemetry libraries to assist with instrumentation for these languages. For more information, see the OpenTelemetry documentation. Cloud Pak for Integration capabilities such as IBM MQ and IBM App Connect do not yet log trace spans to Jaeger, so they are excluded from the analysis.

To create a minimal instance of the API Management capability, complete the following steps:

1. Log in to IBM Cloud Pak Platform UI.

1. Click Integration Capabilities > Create capability.

2. Select API management and click Next. If the API management tile is not available, make sure you have installed the IBM API Connect operator. See Installing the operators using the Openshift console for more information.

3. Select API Endpoints - One Node - Minimum, then click Next.

4. Specify a name.

5. Accept the license and choose the appropriate value from the dropdown. For more information on the license, see API Connect licenses.

6. To add additional functionality to the instance that supports AI Test Generation, go to [ and enable Advanced settings.

7. Configure the API gateway so it logs trace spans to Jaeger

   1. Select Gateway subsystem.

   2. Under Jaeger Tracing, scroll down and set the Jaeger collector endpoint to be the collector endpoint of the Jaeger instance you created in Installing Jaeger in your Production environment. The Jaeger agent deployed with the API Connect gateway talks to the collector service on port 14250 (HTTPS gRPC), so the appropriate endpoint based on the example instance name jaeger-production would be:

      jaeger-production-collector-headless:14250

      • Set the TLS Secret name to be the secret you identified in Installing Jaeger in your Production environment.

      • Set the TLS Server name to be the full cluster local DNS name for the collector service. Based on the example instance name jaeger-production in the project my-project the server name would be:

        jaeger-production-collector-headless.my-project.svc.cluster.local

8. Click Create to deploy the API Management capability.

For additional information on how to deploy and configure API management, see:

• API management deployment

• Managing APIs with API Connect

To deploy the Data service that serves trace and payload data to the analyzer:

1. Log in to IBM Cloud Pak Platform UI.

2. Follow the instructions from Installing API Management through step 9.

3. Select Management subsystem

4. Enable the AI data service.

5. Set the Jaeger endpoint for API data. The endpoint you specified for the Gateway is the collector endpoint, and this is the query endpoint that provides access to the collected traces. Specify the name of the service you created earlier, on port 16685 (older versions of Jaeger, pre 1.22, use 16686):

   jaeger-production-query-api:16685

   Alternatively, you can express the configuration as a Custom Resource definition. Select the YAML view and copy and paste the following example:

   apiVersion: apiconnect.ibm.com/v1beta1
   kind: APIConnectCluster
   metadata:
     name: apiconnect-production
     labels:
       app.kubernetes.io/instance: apiconnect
       app.kubernetes.io/managed-by: ibm-apiconnect
       app.kubernetes.io/name: apiconnect-minimum
   spec:
     license:
       accept: true
       use: nonproduction
       license: <my_license>
     profile: n1xc7.m48
     version: "10.0.4"
     gateway:
       jaegerTracing:
         jaegerCollectorEndpoint: "jaeger-production-collector-headless:14250"
         tls:
           collectorEndpoint:
             secretName: jaeger-production-collector-headless-tls
             serverName: jaeger-production-collector-headless.my-project.svc.cluster.local
     management:
       aiDataService:
         enabled: true
         jaegerEndpoint: "jaeger-production-query-api:16685"

6. Click Create to deploy the API management capability, and the AI Test Generation Data service, into your project.

To enable AI insight in a test suite (at the end of this procedure), you must provide details about the AI Test generation data service deployed to your Production environment.

To get those details, run the following commands using the OpenShift CLI. In each step, replace:

• apiconnect-production with the actual name of your API management instance.

• my-project with the name of the project which contains that instance.

Data service endpoint:

oc get route -n my-project --selector app.kubernetes.io/part-of=apiconnect-production -o json | \
        jq -r '.items[] | select(.metadata.name|endswith("-api-data-service")) | ("https://" + .spec.host)'

Data service CA certificate:

oc get route -n my-project --selector app.kubernetes.io/part-of=apiconnect-production -o json | \
        jq -r '.items[] | select(.metadata.name|endswith("-api-data-service")) | .spec.tls.caCertificate'

Data service API key:

oc get secret -n my-project --selector app.kubernetes.io/part-of=apiconnect-production -o json | \
        jq -r '.items[] | select(.metadata.name|endswith("-data-service-api-key")) | .data.API_KEY' | \
        base64 --decode

To install Jaeger in the Test environment:

1. Follow the instructions for Installing Jaeger in the OpenShift Container Platform documentation. You can install Jaeger with or without Service Mesh, depending on your requirements.

   Assuming an example name of jaeger-test for your Jaeger instance, the installation should create the following additional resources, which you will need when configuring the API management instance later:

   • A service called jaeger-test-collector-headless.

   • A secret called jaeger-test-collector-headless-tls.

2. Create a service called jaeger-test-query-api that will expose the test trace data for analysis. Under Installing Jaeger in your Production environment, see "Creating the Jaeger Query service" for the procedure.

3. IBM recommends creating a network policy that limits access to only the AI Test Generation Data service deployed in the same project. Under Installing Jaeger in your Production environment, see "Creating a Jaeger network policy" for more information.

4. To improve security for the Jaeger Query gRPC interface follow these steps.

You must deploy an instance of your API into the Test environment to execute test cases (under the control of the API management capability) and it must be configured to log trace spans to the Jaeger collector you created in the previous step. How you deploy your API is beyond the scope of this document, but for general guidance refer to Deploying your API.

To create a minimal instance of the API Management capability, complete the following steps:

1. Log in to Platform UI.

2. Click Integration Capabilities > Create capability.

3. Select API management, then click Next. If the API management tile is not enabled, make sure you have installed the IBM API Connect operator. See Installing the operators using the Openshift console for more information.

4. Select API Endpoints - One Node - Minimum and click Next.

5. Specify a name.

6. Accept the license and choose the appropriate value from the dropdown. For more information on the license, see API Connect licenses.

To configure the API gateway so it logs trace spans to Jaeger:

1. Go to Gateway subsystem

2. Under Jaeger Tracing, scroll down and set the Jaeger collector endpoint to be the collector endpoint of the Jaeger instance you created in Installing Jaeger in your Test environment. The Jaeger agent deployed with the API Connect gateway talks to the collector service on port 14250 (HTTPS gRPC), so the appropriate endpoint based on the example instance name jaeger-test would be:

   jaeger-test-collector-headless:14250

3. Set the TLS Secret name to be the secret you created in Installing Jaeger in your Test environment.

4. Set the TLS Server name to be the full cluster local DNS name for the collector service. Based on the example instance name jaeger-test in the project my-project the server name would be:

jaeger-test-collector-headless.my-project.svc.cluster.local

To enable Test & Monitor with AI insight:

1. Go to Management subsystem

2. Scroll down to find Test and Monitor and click Enable Test and Monitor

3. Click Enable AI test insight

4. Set the Jaeger endpoint for API data. The endpoint you specified for the Gateway is the collector endpoint, and this is the query endpoint that provides access to the collected traces. Specify the name of the service you created earlier, on port 16685 (older versions of Jaeger, pre 1.22, use 16686). For example:

jaeger-test-query-api:16685

Alternatively, you can express the configuration as a Custom Resource definition. Click the YAML view and copy and paste the following exam:

apiVersion: apiconnect.ibm.com/v1beta1
kind: APIConnectCluster
metadata:
  name: apiconnect-test
  labels:
    app.kubernetes.io/instance: apiconnect
    app.kubernetes.io/managed-by: ibm-apiconnect
    app.kubernetes.io/name: apiconnect-minimum
spec:
  license:
    accept: true
    use: nonproduction
    license: MY-LICENSE
  profile: n1xc7.m48
  version: "10.0.4"
  gateway:
    jaegerTracing:
      jaegerCollectorEndpoint: "jaeger-test-collector-headless:14250"
      tls:
        collectorEndpoint:
          secretName: jaeger-test-collector-headless-tls
          serverName: jaeger-test-collector-headless.my-project.svc.cluster.local
  management:
    testAndMonitor:
      enabled: true
      aiEnabled: true
      jaegerEndpoint: "jaeger-test-query-api.cp4i.svc:16685"

1. Click Create to deploy the API management capability, with Test and Monitor and AI insight, into your project.

For additional information on how to deploy and configure API management, see:

• API management deployment

• Managing APIs with API Connect.

To further configure the instance to support Test and Monitor with AI insight, enable Advanced settings.

In the API Manager UI:

1. Select Develop APIs and products.

2. Open your API.

3. Select the Test tab.

4. Send some test requests.

If you see the correct responses, this confirms that your API is operating through the gateway.

In the Red Hat OpenShift console:

1. Click Networking then Routes.

2. Select the project for your environment.

3. Search for your Jaeger instance (jaeger-production or jaeger-test).

4. Click the Location link to open the Jaeger UI.

5. Set the Service to apiconnect.

6. Click Find Traces.

If you see traces corresponding to your test API calls, this confirms that the gateway and your API are logging trace spans to Jaeger.