IBM Support

QRadar: Apps stuck in UPGRADING status after upgrade attempt

Troubleshooting


Problem

Upgrading all the apps through the QRadar Assistant app at once fails. The apps get stuck in UPGRADING state.

Symptom

Cause

This issue happens when the zip_source_file_path is wrong for apps stuck in upgrading, this information can be confirmed by using the app manager utility (qappmanager).

Diagnosing The Problem

Use these steps to confirm if the apps have the right zip source file path:
  1. SSH into the QRadar console.
  2. Run the next command to open the app manager utility to validate the app status: 
    /opt/qradar/support/qappmanager
    Result example: 
     ID   | Name                                                         | Version | Status    | Installed        | Memory | Instances | SIO | MTS | Errors                                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1255 | IBM QRadar Operations                                        | 2.3.0   | UPGRADING | 2020-06-10 14:28 |    200 |         1 | f   | f   |                                                
 1259 | QRadar App Editor                                            | 3.0.0   | UPGRADING | 2020-06-10 14:40 |    200 |         1 | f   | f   |                                                
 1551 | IBM QRadar DNS Analyzer                                      | 2.0.0   | UPGRADING | 2020-09-17 15:31 |   4096 |         1 | f   | f   |                                                
 1901 | QRadar Assistant                                             | 3.5.2   | UPGRADING | 2021-06-09 19:58 |    600 |         1 | t   | t   |                                                
 2001 | Recorded Future for IBM QRadar – QRadar v7.3.3FP6+/7.4.1FP2+ | 3.0.0   | UPGRADING | 2021-09-23 18:52 |    400 |         1 | f   | t   | Warning: Field "RF_URL" does not exist in catalogs events, flows, simarc, statistics. This may indicate application components will not work. Continuing with application creation.
 2101 | User Analytics                                               | 4.1.6   | UPGRADING | 2022-02-08 17:43 |   1000 |         1 | f   | t   |                                                                          
 2103 | Machine Learning Analytics                                   | 4.1.6   | UPGRADING | 2022-02-17 21:35 |  40000 |         1 | f   | t   |                                                
....
    The Status column shows that the apps are in UPGRADING state.
  3. Run the next command to confirm the zip_source_file_path: 
    psql -U qradar -c "select id,name,zip_source_file_path from installed_application;"
    Output example: 
    psql -U qradar -c "select id,name,zip_source_file_path from installed_application;"
  id  |                             name                             |   zip_source_file_path
------+--------------------------------------------------------------+---------------------------
 2101 | User Analytics                                               | /storetmp/AppFW_2101.zip
 2103 | Machine Learning Analytics                                   | /storetmp/AppFW_2103.zip
 1259 | QRadar App Editor                                            | /storetmp/AppFW_1259.zip
 2001 | Recorded Future for IBM QRadar – QRadar v7.3.3FP6+/7.4.1FP2+ | /storetmp/AppFW_2001.zip
 1255 | IBM QRadar Operations                                        | /storetmp/AppFW_1255.zip
 1551 | IBM QRadar DNS Analyzer                                      | /storetmp/AppFW_1551.zip
 1901 | QRadar Assistant                                             | /storetmp/AppFW_1901.zip
    The path /storetmp/ is a wrong zip_source_file_path. The right path is /store/qapp/.

    Output example of nonaffected apps: 
    psql -U qradar -c "select id,name,zip_source_file_path from installed_application;"
  id  |                             name                             |   zip_source_file_path
------+--------------------------------------------------------------+---------------------------
 1202 | Palo Alto Networks App for QRadar                            | /store/qapp/1202/1202.zip
 1401 | Cloud Visibility                                             | /store/qapp/1401/1401.zip
 1260 | QRadar Use Case Manager                                      | /store/qapp/1260/1260.zip
 2151 | Deployment Intelligence                                      | /store/qapp/2151/2151.zip
 1251 | CarbonBlack                                                  | /store/qapp/1251/1251.zip
 1051 | ThreatConnect                                                | /store/qapp/1051/1051.zip

    Result
    Administrators can confirm if the stuck apps have the right zip source file path in the database. The right path is /store/qapp/.

Resolving The Problem

Important disclaimers:
  • Read all the steps before they are applied on a production environment. 
  • The following steps require advanced knowledge. Administrators who feel uncomfortable with this procedure or need assistance, contact QRadar Support.
Use these steps to correct the zip source file path in the database:
  1. SSH into the QRadar console.
  2. Create a backup directory in case it doesn't exist already: 
    mkdir -p /store/IBM_Support/
  3. Run the following two commands to create a backup for the tables: 
    pg_dump -U qradar -t installed_application > /store/IBM_Support/installed_application.sql
pg_dump -U qradar -t installed_application_instance > /store/IBM_Support/installed_application_instance.sql
  4. Update the app status for all the apps to COMPLETED by using the following command: 
    psql -U qradar -c "update installed_application set status='COMPLETED';"
  5. Stop all the apps, run the following command: 
    psql -U qradar -c "update installed_application_instance set status='STOPPED', task_status='COMPLETED';"
  6. Use the following command to find the ID for each app: 
    psql -U qradar -c "select id,name,zip_source_file_path from installed_application;"
    Output example: 
    psql -U qradar -c "select id,name,zip_source_file_path from installed_application;"
  id  |                             name                             |   zip_source_file_path
------+--------------------------------------------------------------+---------------------------
 2101 | User Analytics                                               | /storetmp/AppFW_2101.zip
 2103 | Machine Learning Analytics                                   | /storetmp/AppFW_2103.zip
 1259 | QRadar App Editor                                            | /storetmp/AppFW_1259.zip
 2001 | Recorded Future for IBM QRadar – QRadar v7.3.3FP6+/7.4.1FP2+ | /storetmp/AppFW_2001.zip
 1255 | IBM QRadar Operations                                        | /storetmp/AppFW_1255.zip
 1551 | IBM QRadar DNS Analyzer                                      | /storetmp/AppFW_1551.zip
 1901 | QRadar Assistant                                             | /storetmp/AppFW_1901.zip
  7. Change the zip_source_file_path value for the correct one by using the following command, replace <app_id> with the app ID seen in the previous command, this information has to be added three times on the command: 
    psql -U qradar -c "update installed_application set zip_source_file_path='/store/qapp/<app_id>/<app_id>.zip' where ID=<app_id>;"
    Command example: 
    psql -U qradar -c "update installed_application set zip_source_file_path='/store/qapp/2101/2101.zip' where ID=2101;"
    Repeat the command for each app ID.
  8. Open the app manager utility, run the following command: 
    /opt/qradar/support/qappmanager
  9. Enter the number "23" to access the section on the app manager utility to start apps:
    image-20221031150115-1
  10. Enter the ID of a valid token with admin role:
    image-20221031150244-3
  11. Check the ID column and enter the ID:
    image-20221102140731-1
  12. Repeat the same process with all the apps. Enter again 23, then select the next app ID. It is not necessary to select the token:
    image-20221102141258-2

    Result
    The Status for all the apps changes from UPGRADING to RUNNING.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSV4BL","label":"IBM QRadar"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
22 August 2023

UID

ibm16829111

Page Feedback