Troubleshooting integration servers and integration runtimes in the App Connect Dashboard

To resolve this error, complete the following steps:

1. Update the custom resource settings of the integration runtime to indicate what type of flows are contained in the BAR file. Set value to true or false as follows to indicate that the flows are built in the IBM App Connect Enterprise Toolkit (toolkitFlow), or that they are API flows (designerAPIFlow) or event-driven flows (designerEventFlow) that are authored in App Connect Designer.

   spec:
     flowType:
       toolkitFlow: value
       designerAPIFlow: value
       designerEventFlow: value

   For more information, see Integration runtime: Updating the custom resource settings for an instance and Integration runtime: Custom resource values.

2. Repeat the steps that you completed earlier to create the integration runtime.
3. Optional: Raise a support ticket (or case) and provide a copy of your Operator log and a copy of the BAR file if you want IBM to investigate the root cause of the problem.

About this task

If a deployed BAR file for a Designer or Toolkit integration contains a definition that is complex or resource-intensive, on startup of the integration server or integration runtime, you might notice that its pod continually restarts.

To resolve this issue, check the CPU and memory usage of the pod to establish whether the default minimum limits need to be increased. For example, to check the resource usage from the Red Hat® OpenShift® web console, you can view the integration server or integration runtime pod details from the Details tab under Workloads > Pods, and then click the usage graph to view the metrics.

If the pod is close to or exceeding its resource limits, increase the limits to stop the restarts or to stop the pod from crashing.

From the App Connect Dashboard, edit the integration server or integration runtime definition to adjust the resource limits for the pod's runtime container.

1. Complete the relevant steps:
   • Integration server:
     1. From the Servers page, open the options menu on the integration server tile and then click Edit.
     2. From the Edit Instance page, either use the Common settings view (with Advanced settings enabled) or the YAML editor view to increase the limits for the runtime container. You can update the following parameters or fields:
        • spec.pod.containers.runtime.resources.limits.cpu
        • spec.pod.containers.runtime.resources.limits.memory

        For information about these parameters, see Integration Server reference: Custom resource values.

        
   • Integration runtime:
     1. From the Runtimes page, open the options menu on the integration runtime tile and then click Edit.
     2. From the Edit Instance page, either use the Common settings view (with Advanced settings enabled) or the YAML editor view to increase the limits for the runtime container. You can update the following parameters or fields:
        • spec.template.spec.containers[].name (set to runtime)
        • spec.template.spec.containers[].resources.limits.cpu
        • spec.template.spec.containers[].resources.limits.memory

        For information about these parameters, see Integration Runtime reference: Custom resource values.

        
2. Click Update.
   Tip: You can also edit the resource limits from the Red Hat OpenShift web console or CLI. If using the CLI, you can use commands such as oc patch integrationserver or oc edit integrationserver for integration servers, or oc patch integrationruntime or oc edit integrationruntime for integration runtimes.

To check the deployment status of an integration server or integration runtime, complete the following steps.

1. After you deploy an integration server or integration runtime to the App Connect Dashboard, you can check whether the deployment was successful by running the following command in the CLI.

   oc get pods

   The command returns a list of pods in the namespace. Integration server pods are named in the format integrationServerName-is-generated_characters, and integration runtime pods are named in the format integrationRuntimeName-ir-generated_characters.
2. Check the STATUS field for issues. A pod that is working correctly moves from a status of Pending to Container creating to Running.
   

   A failed deployment might be due to an issue with the configuration that you specified or because you ran out of cluster resources.

3. To get more detailed information about that pod, run the following command:

   oc describe pod podName

   

To retrieve user logs for an integration server, complete the following steps:

1. Retrieve the list of pods in the namespace by running the following command:

   oc get pods

2. Identify the pod name for the integration server or integration runtime whose logs you want to retrieve and then run the following command. (The pod name is in the format integrationServerName-is-generated_characters or integrationRuntimeName-ir-generated_characters.)

   oc describe pod podName

3. Locate the container name in the output and then run the following command to obtain the App Connect user logs for the pod:

   oc logs podName -c container_name

   The logs are written to a standard output (stdout) stream in the terminal window as shown in the following example.

   Warning:

   If you are running on Red Hat OpenShift, do not stream logs with large volumes of log messages to standard output. Doing so can cause pod logging to hang. The pod might appear to not be running, but the underlying runtimes will remain fully operational.

   In exceptional circumstances, you can log longer lines by setting the ACE_LOGGING_BUFFER environment variable to an integer, to customize the log buffer in KB. The advice is to set this value to twice the size of the largest message that you want to emit. For example, for a 2 MB message, set the ACE_LOGGING_BUFFER environment variable to 4096. This workaround is considered a temporary solution and must not be permanently configured on your system. For debugging purposes, use other techniques such as trace nodes. Also ensure that any logging application in your cluster (for example, the ELK Stack) is appropriately configured to allow larger messages to be consumed.

   The following example shows how to set the log buffer in an integration server custom resource.

   spec:
     env:
     - name: ACE_LOGGING_BUFFER
       value: '4096'

   The following example shows how to set the log buffer in an integration runtime custom resource.

   spec:
     template:
       spec:
         containers:
           - name: runtime
              env:
              - name: ACE_LOGGING_BUFFER
                value: '4096'

   C:\WINDOWS\system32>oc logs https-basic-auth-is-5d646b9844-c6h85 -c https-basic-auth
   2020-07-06T10:16:01.450Z Image created: 2020-06-12T03:04:42+00:00
   2020-07-06T10:16:01.450Z Image revision: Not specified
   2020-07-06T10:16:01.450Z Image source: Not specified
   2020-07-06T10:16:01.539Z ACE version: 11009
   2020-07-06T10:16:01.539Z ACE level: S000-L200527.16701
   2020-07-06T10:16:01.539Z ACE build type: Production, 64 bit, amd64_linux_2
   2020-07-06T10:16:01.539Z Checking for valid working directory
   2020-07-06T10:16:01.539Z Checking if work dir is already initialized
   2020-07-06T10:16:01.539Z Checking for contents in the work dir
   2020-07-06T10:16:01.539Z Work dir initialization complete
   2020-07-06T10:16:01.539Z SetupConfigurationsFiles
   2020-07-06T10:16:01.539Z SetupConfigurationsFiles - configuration names: https-keystore-1.p12,https-policyproject-1,https-serverconf-1,https-setdbparms-1
   2020-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-keystore-1.p12 type: keystore
   2020-07-06T10:16:01.600Z constructKeyStoreOnFileSystem - Keystore name: https-keystore-1.p12
   2020-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-policyproject-1 type: policyproject
   2020-07-06T10:16:01.600Z ConstructPolicyProjectOnFileSystem
   2020-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-serverconf-1 type: serverconf
   2020-07-06T10:16:01.600Z constructServerConfYamlOnFileSystem
   2020-07-06T10:16:01.600Z ConstructConfigurationsOnFileSystem - configuration name: https-setdbparms-1 type: setdbparms
   2020-07-06T10:16:01.600Z ExecuteSetDbParms
   2020-07-06T10:16:01.600Z ExecuteSetDbParms - execute line 0 with number of args: 3
   2020-07-06T10:16:01.685Z ExecuteSetDbParms - execute line 1 with number of args: 3
   2020-07-06T10:16:01.768Z Performing initial configuration of integration server
   2020-07-06T10:16:01.768Z Getting configuration from content server
   2020-07-06T10:16:01.768Z Using ca file /home/aceuser/ssl/cacert.pem
   2020-07-06T10:16:01.798Z Configuration pulled from content server successfully
   2020-07-06T10:16:01.798Z Processing configuration in folder bars
   BIP8071I: Successful command completion.
   2020-07-06T10:16:01.892Z Processing configuration in folder webusers
   2020-07-06T10:16:01.906Z+00:00 Handling webusers configuration
   2020-07-06T10:16:02.074Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:02.269Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:02.421Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:02.580Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:02.733Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:02.888Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:02.895Z+00:00 Creating admin user ibm-ace-dashboard-admin
   2020-07-06T10:16:03.514Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:03.521Z+00:00 Creating operator user ibm-ace-dashboard-operator
   2020-07-06T10:16:04.289Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:04.296Z+00:00 Creating editor user ibm-ace-dashboard-editor
   2020-07-06T10:16:04.914Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:04.920Z+00:00 Creating audit user ibm-ace-dashboard-audit
   2020-07-06T10:16:05.678Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:05.685Z+00:00 Creating viewer user ibm-ace-dashboard-viewer
   2020-07-06T10:16:06.303Z+00:00 BIP8071I: Successful command completion.
   2020-07-06T10:16:06.303Z Enabling metrics in server.conf.yaml
   2020-07-06T10:16:06.304Z Metrics enabled in server.conf.yaml
   2020-07-06T10:16:06.304Z Initial configuration of integration server complete
   2020-07-06T10:16:06.304Z Discovering override ports
   2020-07-06T10:16:06.519Z Successfully discovered override ports
   2020-07-06T10:16:06.519Z Starting integration server
   2020-07-06T10:16:06.519Z Waiting for integration server to be ready
   2020-07-06T10:16:06.522Z Integration server not ready yet
   .....2020-07-06 10:16:06.703125: .2020-07-06 10:16:06.703346: Integration server 'https-basic-auth' starting initialization; version '11.0.0.9' (64-bit)
   ........................................2020-07-06 10:16:09.020610: About to 'Initialize' the deployed resource 'CustomerDatabaseV1' of type 'RestAPI'.
   2020-07-06T10:16:11.526Z Integration server not ready yet
   2020-07-06T10:16:16.576Z Integration server not ready yet
   2020-07-06 10:16:18.227306: About to 'Start' the deployed resource 'CustomerDatabaseV1' of type 'RestAPI'.
   An https endpoint was registered on port '7843', path '/customerdb/v1*'.
   2020-07-06 10:16:18.253094: The HTTP Listener has started listening on port '7843' for 'https' connections.
   2020-07-06 10:16:18.253180: Listening on HTTP URL '/customerdb/v1*'.
   Started native listener for HTTPS input node on port 7843 for URL /customerdb/v1*
   2020-07-06 10:16:18.253436: Deployed resource 'gen.CustomerDatabaseV1' (uuid='gen.CustomerDatabaseV1',type='MessageFlow') started successfully.
   ..2020-07-06 10:16:18.928608: IBM App Connect Enterprise administration security is authentication, authorization file. 
   2020-07-06 10:16:18.939272: The HTTP Listener has started listening on port '7600' for 'RestAdmin http' connections.
   
   2020-07-06 10:16:18.942780: Integration server has finished initialization.
   2020-07-06T10:16:21.585Z Integration server is ready
   2020-07-06T10:16:21.585Z Gathering Metrics...
   2020-07-06T10:16:21.585Z Starting metrics gathering
   2020-07-06T10:16:21.585Z Processing metrics...
   2020-07-06T10:16:21.585Z ACE_ADMIN_SERVER_SECURITY is false
   2020-07-06T10:16:21.585Z Connecting to ws://localhost:7600/ for statistics
   2020-07-06T10:16:21.893Z Connecting to ws://localhost:7600/ using session cookie
   

To aid with problem determination and troubleshooting, you can enable and then download user or service trace on a deployed integration server or integration runtime. Enabling trace is useful if you cannot get enough information about a particular problem from the entries that are available in the log.

You can enable and manage trace as follows:

• From the Red Hat OpenShift web console or CLI, or the CLI for a Kubernetes environment, you can start (or enable) a trace on a deployed integration server or integration runtime by creating a trace object, and stop (or disable) a trace by deleting the trace object. You can also start and stop trace by using the server.conf.yaml file or an environment variable. For more information, see Trace reference.
• From an App Connect Dashboard instance, you can start a trace, stop a trace, download the trace log files, and clear the trace information from the logs by using the options menu on an integration server tile on the Servers page or an integration runtime tile on the Runtimes page. You can also start and stop trace by using the server.conf.yaml file or an environment variable. For more information, see Enabling and managing trace for a deployed integration server or Enabling and managing trace for a deployed integration runtime.
• If you need to increase the download speed of the trace log files for an integration server or integration runtime, you can choose to allocate more CPU by updating the integration server or integration runtime custom resource. For more information, see Integration Server reference: Updating the custom resource settings for an instance or Integration Runtime reference: Updating the custom resource settings for an instance.
• If an integration server or integration runtime container is crashing, you can use an environment variable to prevent the container from stopping so that the trace can be collected as normal. For more information, see Collecting trace from an integration server or integration runtime container that is crashing.

To skip the validation checks and prevent the ValidatingAdmissionWebhook error, complete the following steps:

1. Edit the integration server or integration runtime custom resource (CR) settings to add the following annotation to the metadata block while in YAML view:

   metadata:
     annotations:
       appconnect.ibm.com/webhook-barcheck: NONE

   You can do so from the Red Hat OpenShift web console or CLI, as described in Integration Server reference: Updating the custom resource settings for an instance and Integration Runtime reference: Updating the custom resource settings for an instance.

   Note:

   • This annotation setting is applicable only for integration servers or integration runtimes with a spec.version value that resolves to 12.0.9.0-r3 or later.
   • When this annotation is applied, the specified BAR files (in spec.barURL) are not validated to check whether they can be reached. If any BAR file is unreachable with a 404 status code, the integration server or integration runtime fails to start.
2. Save your changes to the CR.