IBM Support

QRadar: How to use Recon to troubleshoot QRadar applications

Troubleshooting


Problem

How do you use recon ps to view logs for QRadar applications?

Environment

In QRadar V7.3.2 and V7.3.3, the support tool for application troubleshooting is named recon. Recon replaces the QRadar V7.3.1 and earlier troubleshooting application named qapp_utils730.py. 

Resolving The Problem

Recon is a tool designed to aid the troubleshooting of containers and container management on the QRadar Console or App Host. Recon features multiple commands for this purpose. Similar to tools like ps and docker ps, recon ps allows you see an overview of what containers are currently running on the system and available properties for them. Recon uses integration with the DrQ tool to diagnose problems that can occur and show a brief chart. This command also lists possible remediation for each potential error found. Due to API limitations, Recon does not know what managed host it's being run on. Recon lists what applications exist according to the QRadar API, and also what containers are currently running on the current host. They might not be the same set. When given an app ID, or a combination of workload, service, and container, recon connect allows you to run a specific command inside of a specific container.

Procedure
  1. Using SSH, log in to the QRadar Console as the root user.
    Note: The following commands must be run on the host where the Applications are running. If you are running an App Host, you need to SSH to it to run the commands.  
  2. To locate the application ID for your app, type the following command.
    /opt/qradar/support/recon ps
    A list of installed applications and their App-ID values are output to the screen. The App-ID is a unique numeric value. Administrators can use the numeric App-ID value to connect to the container for a specific application.

    If no issues are detected, the recon command output might look like the following example:
    /opt/qradar/support/recon ps
    
    App-ID  Name                            Managed Host ID Workload ID             Service Name    AB      Container Name  CDEGH   Port    IJKL
    1151    QRadar Log Source Management    53              apps                    qapp-1151       ++      qapp-1151       +++++   5000    ++++
    1201    User Analytics                  53              apps                    qapp-1201       ++      qapp-1201       +++++   5000    ++++
    1102    QRadar Assistant                53              apps                    qapp-1102       ++      qapp-1102       +++++   5000    ++++
    1103    QRadar Use Case Manager         53              apps                    qapp-1103       ++      qapp-1103       +++++   5000    ++++
    
    Legend:
    
    Symbols:
    n - Not Applicable
    - - Failure
    * - Warning
    + - Success
    
    Checks:
    Service:
    A - Service exists in the workload file
    B - Service is set to started
    
    Container:
    C - Container is in ConMan workload file
    D - Container environment file exists
    E - Container image is in si-registry
    G - Container Systemd Units are started
    H - Container exists and is running in Docker
    
    Port:
    I - Container IP are in firewall main filter rules
    J - Container IP and port is in iptables NAT filter rules
    K - Container port has routes through Traefik
    L - Container port is responsive on debug path
  3. To connect to the app container, type the following command.
    /opt/qradar/support/recon connect 1005
    A shell is opened to the application's container. Administrators can browse this directory to review files, logs, or configurations for the application.
     
  4. To review an application log for connection errors, type the following command.
    less /store/docker/volumes/qapp-<appID>/log/app.log 
  5. If you want to do a test connection from QRadar resilient Application to another server (like proxy, auth server) to check the reachability, run the following command: 
    sh-4.4$ curl -v telnet://X.X.X.11:443
    Example output.
    * Rebuilt URL to: telnet://X.X.X.11:443/ 
    *   Trying X.X.X.11... 
    * TCP_NODELAY set 
    * Connected to X.X.X.11 (X.X.X.11) port 443 (#0) 
Note:  Do not use the telnet command inside the application container to perform the reachability test as the telnet command is not available inside the application container.

Possible errors returned.
If the following message is returned, confirm that IBM QRadar Console UI Login page is accessible.
Unable to communicate with API. Received error: An API error occurred. The HTTP status code of 404 was returned. The API returned the body of:
&ps.StatusResult{Check:(*ps.StatusCheck)(nil), Message:"error getting response for url https://qradar-qr750.local:9000/v1/api/workloads: Get https://qradar-qr750.local:9000/v1/api/workloads: dial tcp xxx.xxx.xxx.xxx:9000: connect: connection refused", Remediation:"", Value:1}

App-ID  Name    Managed Host ID Workload ID                             Service Name            Container Name          Port
                0               Failed to decode workloads      -                                                       0

Legend:

Symbols:
n - Not Applicable
- - Failure
* - Warning
+ - Success

Checks:
Service:
A - Service exists in the workload file
If an application fails to start the following message is displayed along with the remediation steps required.
/opt/qradar/support/recon ps
App-ID  Name                            Managed Host ID Workload ID             Service Name    AB      Container Name  CDEGH   Port    IJKL
1151    QRadar Log Source Management    53              apps                    qapp-1151       ++      qapp-1151       +++++   5000    ++++
1201    User Analytics                  53              apps                    qapp-1201       ++      qapp-1201       +++++   5000    ++++
1103    QRadar Use Case Manager         53              apps                    qapp-1103       +-      qapp-1103       +-+--   5000    ----
1102    QRadar Assistant                53              apps                    qapp-1102       ++      qapp-1102       +++++   5000    ++++

Legend:

Symbols:
n - Not Applicable
- - Failure
* - Warning
+ - Success

Checks:
Service:
A - Service exists in the workload file
B - Service is set to started

Container:
C - Container is in ConMan workload file
D - Container environment file exists
E - Container image is in si-registry
G - Container Systemd Units are started
H - Container exists and is running in Docker

Port:
I - Container IP are in firewall main filter rules
J - Container IP and port is in iptables NAT filter rules
K - Container port has routes through Traefik
L - Container port is responsive on debug path


Remediations:

B on Service qapp-1103:
The application is not started.
Go to https://ibm.biz/recon_doc for application troubleshooting information. Choose your version of QRadar from the 'Change version or product' drop-down menu.


D on Container qapp-1103:
Config file is missing.
Follow these steps to resync ConMan.
1. Put ConMan into debug mode by typing 'conman-support set-config -p CONMAN_LOG_LEVEL -v DEBUG'
2. Remove existing files using 'rm /etc/conman/container@*'
3. Restart ConMan by typing 'conman-support restart'
4. Review /var/log/qradar/conman.log for errors.


G on Container qapp-1103:
The config file for container qapp-1103 was not found at /etc/conman/container@14814818600457963407.
Follow these steps to resync ConMan.
1. Put ConMan into debug mode by typing 'conman-support set-config -p CONMAN_LOG_LEVEL -v DEBUG'
2. Remove existing files using 'rm /etc/conman/container@*'
3. Restart ConMan by typing 'conman-support restart'
4. Review /var/log/qradar/conman.log for errors.


H on Container qapp-1103:
The app was not found in docker.
1. Put conwrap  into debug mode by typing 'conman-support set-config -p CONWRAP_LOG_LEVEL -v DEBUG apps qapp-1103 qapp-1103'.
2. Check the logs at 'journalctl -u [email protected]' for errors.
If the Conwrap vault token is being rejected, check that the token works, is not expired, and that vault-qrd is active.
If the dockerApps network is down:
        a) Bring up the interface in ifconfig by typing 'ifconfig dockerApps up' and review /var/log/messages
        b) Make sure the /etc/docker/network.d/dockerApps.txt file exists.


K on Port 5000:
Unable to connect to the container qapp-1103 through traefik.
1. Ensure that the traefik service is running by typing 'systemctl is-active traefik' and ensure it started without errors by typing 'journalctl -u traefik'.
2. Ensure that iptables has no rules which might be blocking communication to port 14433.
3. Ensure that the traefik certificate and key in /etc/traefik/tls is present and not expired by typing 'openssl x509 -enddate -noout -in /etc/traefik/tls/traefik.cert'.

L on Port 5000:
Unable to connect to the container qapp-1103 on the debug endpoint: /qapp-1103/flask with error: Received an invalid HTTP response from Traefik endpoint on: /qapp-1103/flask/debug
1. Ensure that the traefik service is started by typing 'systemctl is-active traefik'.
2. Ensure that iptables has no rules which might be blocking communication to the container's port 5000.
3. Ensure that the web service inside the container is active by using the recon support tool to enter the container and inspect the supervisord logs in 'recon connect apps qapp-1103 qapp-1103 cat /store/log/'.
4. Ensure that the traefik certificate and key in /etc/traefik/tls is present and not expired by typing 'openssl x509 -enddate -noout -in /etc/traefik/tls/traefik.cert'.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
19 July 2023

UID

ibm11079655