Troubleshooting integration runtimes in the App Connect Dashboard

To resolve this error, complete the following steps:

1. Update the custom resource settings of the integration runtime to indicate what type of flows are contained in the BAR file. Set value to true or false as follows to indicate that the flows are built in the IBM App Connect Enterprise Toolkit (toolkitFlow), or that they are API flows (designerAPIFlow) or event-driven flows (designerEventFlow) that are authored in App Connect Designer.

   spec:
     flowType:
       toolkitFlow: value
       designerAPIFlow: value
       designerEventFlow: value

   For more information, see Integration runtime: Updating the custom resource settings for an instance and Integration runtime: Custom resource values.

2. Repeat the steps that you completed earlier to create the integration runtime.
3. Optional: Raise a support ticket (or case) and provide a copy of your Operator log and a copy of the BAR file if you want IBM to investigate the root cause of the problem.

About this task

If a deployed BAR file for a Designer or Toolkit integration contains a definition that is complex or resource-intensive, on startup of the integration runtime, you might notice that its pod continually restarts.

To resolve this issue, check the CPU and memory usage of the pod to establish whether the default minimum limits need to be increased. For example, to check the resource usage from the Red Hat OpenShift web console, you can view the integration runtime pod details from the Details tab under Workloads > Pods, and then click the usage graph to view the metrics.

If the pod is close to or exceeding its resource limits, increase the limits to stop the restarts or to stop the pod from crashing.

From the App Connect Dashboard, edit the integration runtime definition to adjust the resource limits for the pod's runtime container.

1. Complete the following steps:
   1. From the Runtimes page, open the options menu on the integration runtime tile and then click Edit.
   2. From the Edit Instance page, either use the Common settings view (with Advanced settings enabled) or the YAML editor view to increase the limits for the runtime container. You can update the following parameters or fields:
      • spec.template.spec.containers[].name (set to runtime)
      • spec.template.spec.containers[].resources.limits.cpu
      • spec.template.spec.containers[].resources.limits.memory

      For information about these parameters, see Integration Runtime reference: Custom resource values.

      
2. Click Update.
   Tip: You can also edit the resource limits from the Red Hat OpenShift web console or CLI. If using the CLI, you can use commands such as oc patch integrationruntime or oc edit integrationruntime.

To check the deployment status of an integration runtime, complete the following steps.

1. After you deploy an integration runtime to the App Connect Dashboard, you can check whether the deployment was successful by running the following command in the CLI.

   oc get pods

   The command returns a list of pods in the namespace. Integration runtime pods are named in the format integrationRuntimeName-ir-generated_characters.
2. Check the STATUS field for issues. A pod that is working correctly moves from a status of Pending to Container creating to Running.
   

   A failed deployment might be due to an issue with the configuration that you specified or because you ran out of cluster resources.

3. To get more detailed information about that pod, run the following command:

   oc describe pod podName

   

To retrieve user logs for an integration runtime, complete the following steps:

1. Retrieve the list of pods in the namespace by running the following command:
   

   oc get pods

   

   kubectl get pods

2. Identify the pod name for the integration runtime whose logs you want to retrieve and then run the following command.
   Tip:

   The integration runtime pod name is in the format integrationRuntimeName-ir-generated_characters.

   Similarly, pod names for the Operator and its other instances are in the format ibm-appconnect-operator-generated_characters, designerName-ui-generated_characters, dashboardName-dash-generated_characters, and switchServerName-switch-generated_characters.

   

   oc describe pod podName

   

   kubectl describe pod podName

3. Locate the container name in the output and then run the following command to obtain the App Connect user logs for the pod:
   

   oc logs podName -c container_name

   

   kubectl logs podName -c container_name

   The logs are written to a standard output (stdout) stream in the terminal window as shown in the following example.

   Warning:

   Applicable only to integration runtimes at version 13.0.5.0-r1 or earlier

   If you are running on Red Hat OpenShift, do not stream logs with large volumes of log messages to standard output. Doing so can cause pod logging to hang. The pod might appear to not be running, but the underlying runtimes will remain fully operational.

   In exceptional circumstances, you can log longer lines by setting the ACE_LOGGING_BUFFER environment variable to an integer, to customize the log buffer in KB. The advice is to set this value to twice the size of the largest message that you want to emit. For example, for a 2 MB message, set the ACE_LOGGING_BUFFER environment variable to 4096. This workaround is considered a temporary solution and must not be permanently configured on your system. For debugging purposes, use other techniques such as trace nodes. Also ensure that any logging application in your cluster (for example, the ELK Stack) is appropriately configured to allow larger messages to be consumed.

   The following example shows how to set the log buffer in an integration runtime custom resource.

   spec:
     template:
       spec:
         containers:
           - name: runtime
              env:
              - name: ACE_LOGGING_BUFFER
                value: '4096'

   C:\WINDOWS\system32>oc logs https-basic-auth-is-5d646b9844-c6h85 -c https-basic-auth
   2023-07-06T10:16:01.450Z Image created: 2023-06-12T03:04:42+00:00
   2023-07-06T10:16:01.450Z Image revision: Not specified
   2023-07-06T10:16:01.450Z Image source: Not specified
   2023-07-06T10:16:01.539Z ACE version: 11009
   2023-07-06T10:16:01.539Z ACE level: S000-L200527.16701
   2023-07-06T10:16:01.539Z ACE build type: Production, 64 bit, amd64_linux_2
   2023-07-06T10:16:01.539Z Checking for valid working directory
   2023-07-06T10:16:01.539Z Checking if work dir is already initialized
   2023-07-06T10:16:01.539Z Checking for contents in the work dir
   2023-07-06T10:16:01.539Z Work dir initialization complete
   2023-07-06T10:16:01.539Z SetupConfigurationsFiles
   2023-07-06T10:16:01.539Z SetupConfigurationsFiles - configuration names: https-keystore-1.p12,https-policyproject-1,https-serverconf-1,https-setdbparms-1
   2023-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-keystore-1.p12 type: keystore
   2023-07-06T10:16:01.600Z constructKeyStoreOnFileSystem - Keystore name: https-keystore-1.p12
   2023-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-policyproject-1 type: policyproject
   2023-07-06T10:16:01.600Z ConstructPolicyProjectOnFileSystem
   2023-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-serverconf-1 type: serverconf
   2023-07-06T10:16:01.600Z constructServerConfYamlOnFileSystem
   2023-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-setdbparms-1 type: setdbparms
   2023-07-06T10:16:01.600Z ExecuteSetDbParms
   2023-07-06T10:16:01.600Z ExecuteSetDbParms - execute line 0 with number of args: 3
   2023-07-06T10:16:01.685Z ExecuteSetDbParms - execute line 1 with number of args: 3
   2023-07-06T10:16:01.768Z Performing initial configuration of integration runtime
   2023-07-06T10:16:01.768Z Getting configuration from content server
   2023-07-06T10:16:01.768Z Using ca file /home/aceuser/ssl/cacert.pem
   2023-07-06T10:16:01.798Z Configuration pulled from content server successfully
   2023-07-06T10:16:01.798Z Processing configuration in folder bars
   BIP8071I: Successful command completion.
   2023-07-06T10:16:01.892Z Processing configuration in folder webusers
   2023-07-06T10:16:01.906Z+00:00 Handling webusers configuration
   2023-07-06T10:16:02.074Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:02.269Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:02.421Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:02.580Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:02.733Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:02.888Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:02.895Z+00:00 Creating admin user ibm-ace-dashboard-admin
   2023-07-06T10:16:03.514Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:03.521Z+00:00 Creating operator user ibm-ace-dashboard-operator
   2023-07-06T10:16:04.289Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:04.296Z+00:00 Creating editor user ibm-ace-dashboard-editor
   2023-07-06T10:16:04.914Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:04.920Z+00:00 Creating audit user ibm-ace-dashboard-audit
   2023-07-06T10:16:05.678Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:05.685Z+00:00 Creating viewer user ibm-ace-dashboard-viewer
   2023-07-06T10:16:06.303Z+00:00 BIP8071I: Successful command completion.
   2023-07-06T10:16:06.303Z Enabling metrics in server.conf.yaml
   2023-07-06T10:16:06.304Z Metrics enabled in server.conf.yaml
   2023-07-06T10:16:06.304Z Initial configuration of integration runtime complete
   2023-07-06T10:16:06.304Z Discovering override ports
   2023-07-06T10:16:06.519Z Successfully discovered override ports
   2023-07-06T10:16:06.519Z Starting integration runtime
   2023-07-06T10:16:06.519Z Waiting for integration runtime to be ready
   2023-07-06T10:16:06.522Z Integration runtime not ready yet
   .....2023-07-06 10:16:06.703125: .2023-07-06 10:16:06.703346: Integration runtime 'https-basic-auth' starting initialization; version '12.0.0.9' (64-bit)
   ........................................2023-07-06 10:16:09.020610: About to 'Initialize' the deployed resource 'CustomerDatabaseV1' of type 'RestAPI'.
   2023-07-06T10:16:11.526Z Integration runtime not ready yet
   2023-07-06T10:16:16.576Z Integration runtime not ready yet
   2023-07-06 10:16:18.227306: About to 'Start' the deployed resource 'CustomerDatabaseV1' of type 'RestAPI'.
   An https endpoint was registered on port '7843', path '/customerdb/v1*'.
   2023-07-06 10:16:18.253094: The HTTP Listener has started listening on port '7843' for 'https' connections.
   2023-07-06 10:16:18.253180: Listening on HTTP URL '/customerdb/v1*'.
   Started native listener for HTTPS input node on port 7843 for URL /customerdb/v1*
   2023-07-06 10:16:18.253436: Deployed resource 'gen.CustomerDatabaseV1' (uuid='gen.CustomerDatabaseV1',type='MessageFlow') started successfully.
   ..2023-07-06 10:16:18.928608: IBM App Connect Enterprise administration security is authentication, authorization file. 
   2023-07-06 10:16:18.939272: The HTTP Listener has started listening on port '7600' for 'RestAdmin http' connections.
   
   2023-07-06 10:16:18.942780: Integration runtime has finished initialization.
   2023-07-06T10:16:21.585Z Integration runtime is ready
   2023-07-06T10:16:21.585Z Gathering Metrics...
   2023-07-06T10:16:21.585Z Starting metrics gathering
   2023-07-06T10:16:21.585Z Processing metrics...
   2023-07-06T10:16:21.585Z ACE_ADMIN_SERVER_SECURITY is false
   2023-07-06T10:16:21.585Z Connecting to ws://localhost:7600/ for statistics
   2023-07-06T10:16:21.893Z Connecting to ws://localhost:7600/ using session cookie
   

To aid with problem determination and troubleshooting, you can enable and then download user or service trace on a deployed integration runtime. Enabling trace is useful if you cannot get enough information about a particular problem from the entries that are available in the log.

You can enable and manage trace as follows:

• From the Red Hat OpenShift web console or CLI, or the CLI for a Kubernetes environment, you can start (or enable) a trace on a deployed integration runtime by creating a trace object, and stop (or disable) a trace by deleting the trace object. You can also start and stop trace by using the server.conf.yaml file or an environment variable. For more information, see Trace reference.
• From an App Connect Dashboard instance, you can start a trace, stop a trace, download the trace log files, and clear the trace information from the logs by using the options menu on an integration runtime tile on the Runtimes page. You can also start and stop trace by using the server.conf.yaml file or an environment variable. For more information, see Enabling and managing trace for a deployed integration runtime.
• If you need to increase the download speed of the trace log files for an integration runtime, you can choose to allocate more CPU by updating the integration runtime custom resource. For more information, see Integration Runtime reference: Updating the custom resource settings for an instance.
• If an integration runtime container is crashing, you can use an environment variable to prevent the container from stopping so that the trace can be collected as normal. For more information, see Collecting trace from an integration runtime container that is crashing.

To skip the validation checks and prevent the ValidatingAdmissionWebhook error, complete the following steps:

1. Edit the integration runtime custom resource (CR) settings to add the following annotation to the metadata block while in YAML view:

   metadata:
     annotations:
       appconnect.ibm.com/webhook-barcheck: NONE

   You can do so from the Red Hat OpenShift web console or CLI, as described in Integration Runtime reference: Updating the custom resource settings for an instance.

   Note:

   • This annotation setting is applicable only for integration runtimes with a spec.version value that resolves to 12.0.9.0-r3 or later.
   • When this annotation is applied, the specified BAR files (in spec.barURL) are not validated to check whether they can be reached. If any BAR file is unreachable with a 404 status code, the integration runtime fails to start.
2. Save your changes to the CR.