Troubleshooting
Problem
It is possible to encounter corruption in the UBA postgres database. In this instance, you can re-create the database without having to uninstall and reinstall UBA.
This workaround applies to UBA 4.1.9 and higher.
Symptom
If you see similar errors as in the following examples, you might have a corrupted database.
messages log file:
Jan 24 13:02:38 qradar_apphost_server kernel: [6316612.508367] [<ffffffffa7fc25ad>] oom_kill_process+0x2cd/0x490
Jan 24 13:02:50 qradar_apphost_server kernel: postgres invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
You can check the UBA logs to help you determine whether the UBA database is corrupted. Enter the directory and check the log files:
cd /store/docker/volumes/qapp-1101/logs
app.log:
2023-02-01 16:55:42,222 [DummyThread-4] [ERROR] [APP_ID:1101] [NOT:0000003000] Failed to generate dashboard top panel: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2023-02-01 16:55:42,231 [DummyThread-5] [ERROR] [APP_ID:1101] [NOT:0000003000] Failed to generate system score graph data: FATAL: the database system is in recovery mode
user_import_service.log:
2023-02-01 14:49:07,751 [user_import_service.run] [ERROR] - FATAL: the database system is in recovery mode
Resolving The Problem
You have two options to resolve the problem. Try dropping the data base first, and if that does not work, try method two.
Drop Database
The following procedure drops the entire UBA database from postgresql and restarting the "poll" service re-creates the UBA database. It also maintains your Tuning parameters (Aliases) as well as the LDAP Imports configuration.
Here are the steps to re-create your UBA postgres database:
- From the Console use qappmanager utility to find the App Instance ID for UBA (1101).
/opt/qradar/support/qappmanager
APP INSTANCES (IID=Instance ID, DID=Definition ID, MHN=Managed Host Name, AHT=Application Host Type, SP=Security Profile): IID | DID | Name | Status | Task Status | Installed | MHN | AHT | Memory | SP | Errors -------------------------------------------------------------------------------------------------------------------------------------------------------------- 1101 | 1101 | User Analytics | RUNNING | COMPLETED | 2020-04-01 13:35 | <servername> | LOCAL | 1000 | |
- Log in to the UBA docker container
/opt/qradar/support/recon connect 1101
- Stop all services as follows:
supervisorctl stop flask supervisorctl stop poll supervisorctl stop userimport
- Use psql client to drop the UBA database by using the exact command as follows:
psql -U postgres -c "DROP DATABASE uba"
- Restart services in this specific order.
supervisorctl start poll supervisorctl start flask supervisorctl start userimport
Results
Once completed, execute a manual import of users from all your LDAP connections. If you get an error that prevents you from dropping the database, follow the next procedure.
Unable to drop database
If you get a message when you drop the database that there are users still connected, the database might not be dropped.
You can verify with the following statements:
- Connect to the container from where the apps are running.
/opt/qradar/support/recon connect 1101
- Connect to UBA database to run SELECT statement.
psql -U postgres
SELECT pid,datname,usename,application_name,client_hostname,client_port,backend_start,query_start,query,state FROM pg_stat_activity WHERE state = 'active';
- If so, you have to use the following steps to re-create the database. However, you lose the LDAP Imports that are configured. Make sure you take screenshots unless you already know the settings.
- From the Console Stop the UBA app with qappmanager utility. (Option 24)
/opt/qradar/support/qappmanager
- From where the apps are running:
cd /store/docker/volumes/qapp-1101/
- Type this command exactly as it is written:
mv psql psql_old_bkp
- From the Console Start the UBA app with qappmanager utility. (Option 23)
/opt/qradar/support/qappmanager
Results
Once completed, reconfigure all your LDAP Import connections and import users.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
20 October 2023
UID
ibm17054882