Troubleshooting App Host problems

To verify that the Kubernetes container is running or complete, run the following command on the App Host system.

sudo kubectl get pods -n kube-system

You can also use the kubectl describe command to view information about a pod that failed to run, especially in the "Events" section.

sudo kubectl describe pod/<name> -n <namespace>

For example,

sudo kubectl describe pod/coredns-68ff754cd5-lr5xm -n kube-system

To verify that the App Host containers are running correctly, run the following command on the App Host system.

sudo kubectl get pods -A -l app.kubernetes.io/name=controller

If the containers are not running, use the kubectl describe command to determine the cause of the issue, paying particular attention to the Events section.

sudo kubectl describe pod/<name> -n <namespace> 

The App Hosts have runtime status probes available. When a problem occurs inside the containers, the cluster tries to resolve the problem by restarting the containers. You can check the container restart counter by listing the pods.

sudo kubectl get pods -n <apphost namespace>

If the RESTARTS counter keeps increasing, use the following command to trigger a self-check on each App Host, and collect the logs by using kubectl.

sudo manageAppHost checkconnection [--controller-id <apphost namespace>]

You might have a container problem if you see the following error message.

App is stuck 'Deploying...'

Check that the container for the app was successfully deployed and is running.

sudo kubectl get pods -A -l apps.isc.ibm.com/app-type=app -L app.kubernetes.io/instance

If the container is not running or in error, use the kubectl describe command to try to diagnose the issue.

sudo kubectl describe pod/<name> -n <namespace>

If the App pods are running, check the App Host logic containers. You can access the log information.

sudo kubectl logs deployments/deployment-synchronizer –n <namespace leftcolumn> &&
sudo kubectl logs deployments/deployment-operator –n <namespace leftcolumn>

If the App Host container logs indicate a connection issue, it might be one of the following problems:

• Proxy issue. Follow the instructions in Configuring proxy authentication, starting at step 4.
• DNS issue. Containers that are started in k3s do not inherit the network aliases from /etc/hosts on the App Host system. DNS can be configured by using the manageAppHost dns command. See Create the App Hosts for details.
• Certificate changed. Use the manageAppHost certificate command to download again the SOAR server certificate.

Start by testing the app configuration from the SOAR Platform. Some configuration issues can be discovered by using the app's built-in test feature.

1. Select the app then click the Configuration tab.
2. Select the app.config file from App Settings.
3. Scroll to the end and choose Test Configuration.

   The self-test is individualized for each app.

You can look up the <app deployment> deployment name with the following command.

sudo kubectl get pods -A -l apps.isc.ibm.com/app-type=app -L app.kubernetes.io/instance

If you see the following error, the cert.cer configuration file needs to be the complete certificate chain. When you install an app, the system tries to generate the full chain and upload it. If the cert.cer was modified, restart the app. You can also try setting cafile=false in app.config just as a point of debugging. Modifying app.config and saving it automatically restarts the pod.

Unable to lock /opt/app-root/src/.resilient/resilient_circuits_lockfile: HTTPSConnectionPool(host='example.net', port=443): Max retries exceeded with url: /rest/session (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

For local or test systems, you might see the following error.

HTTPSConnectionPool(host='192.168.1.50', port=443): Max retries exceeded with url: /rest/session (Caused by SSLError(CertificateError("hostname '192.168.1.50' doesn't match 'resilient.localdomain'",),))

Check for errors in the App Manager log. The AppManager is on the SOAR platform. The resilient-app-manager.log can be found in /var/log/resilient-app-manager.

com.ibm.security.apps.manager.client.ManagerClientException: javax.ws.rs.ForbiddenException: HTTP 403 Forbidden

• Check that the App Host and SOAR clocks are not skewed.
• Check whether the pairing key was regenerated on the SOAR platform.

You can use the manageAppHost checkconnection command to test the connection of the paired App Host, as shown in the following example:

[root@apphost ~]# manageAppHost checkconnection
INFO: Checking cluster API connectivity
Select the app host to test:
  [0] All
  [1] 834c437a-20e9-4c96-9c03-c81011035bc9    apphost1          v1.11.1.483
  [2] 87e510ed-dc7b-44d1-bab7-aa1ac91d09d9    apphost2          v1.12.1.530
Choose the app host ID from the list above (0-2):
2
INFO: Using app host 87e510ed-dc7b-44d1-bab7-aa1ac91d09d9 (apphost2)
INFO: Checking app host 87e510ed-dc7b-44d1-bab7-aa1ac91d09d9 (apphost2, version 1.12.1.530)
INFO: Checking controller deployment
INFO: Checking controller functionality
INFO: Checking app deployments
INFO: End of app host check for 87e510ed-dc7b-44d1-bab7-aa1ac91d09d9

>>> App host 87e510ed-dc7b-44d1-bab7-aa1ac91d09d9 (apphost2, version 1.12.1.530) <<<
No problem found.

After you upgrade the App Host software, check the version number.

manageAppHost --version

If the version number did not change, check that the deployment-synchronizer and deployment-operator are running.

sudo kubeclt get pods -n <namespace>

If the deployment-synchronizer is running, check the synchronizer logs.

kubectl logs pod/deployment-synchronizer-<uniqueid> -n <namepsace>

You can check the App Host logs from the SOAR Apps tab.

1. Click Download Logs from the app's Details page.

   The log is the Resilient® Circuits log. It provides the status of the app within the App Host container (shown as the Pod status), and the name of the app repository (shown as image=<name>).

2. Look for errors where Resilient Circuits tries to contact the outside world and Kubernetes networking issues are blocking it.
3. If you see no obvious errors, check the permissions and settings for the app's API key account.

You can also access the log from the App Host system by using the following command.

sudo kubectl logs deployment/<app deployment> -n <namespace>