Troubleshooting
Problem
The size of backups increases, causing high disk usage and system notifications related to disk space issues. How can I diagnose why my backup size fluctuates or suddenly grows in size?
Resolving The Problem
Before you begin
Identify a backup that is larger than expected and one that is with in the size range you expect.
- Use SSH to log in to the QRadar Console as the root user.
- Type the following command:
Example output:ls /store/backup/ -l --block-size=G-rw-r--r-- 1 nobody nobody 9.5G Jul 29 01:57 /store/backup/backup.nightly.x_53.28_07_2021.data.1627523842702.tgz -rw-r--r-- 1 nobody nobody 12G Jul 30 02:09 /store/backup/backup.nightly.x_53.29_07_2021.data.1627610965970.tgz -rw-r--r-- 1 nobody nobody 25G Jul 31 02:20 /store/backup/backup.nightly.x_53.30_07_2021.data.1627698058668.tgz - Observe the size column (example: 9.5G) and the date and time (example: Jul 29 01:57).
Result
The size might gradually increase or suddenly spike. Identify one large file, and one of the size you expect. Note the dates the files were created. If possible, select dates within the same month, as it makes the comparison process easier.
Steps
Investigate the size of the information stored in ariel to determine whether it increased at a similar rate to the backup size or not.
- Use SSH to log in to the QRadar Console as the root user.
- Investigate the flow and event, and payloads and records data in storage. Replace /Year/Month/Day/Hour/Minute with the relevant information. If you want to compare 29 July 2022 to 31 July 2022, you can use /2022/7 to list the data for every day in the month of July. If the selected dates are not within the same month, you must run each command twice to view the directory size for the two months.
- Event records:
du -xhd 1 /store/ariel/events/records/Year/Month/Day/Hour/Minute - Event payloads:
du -xhd 1 /store/ariel/events/payloads/Year/Month/Day/Hour/Minute - Flow records:
du -xhd 1 /store/ariel/flows/records/Year/Month/Day/Hour/Minute - Flow payloads:
du -xhd 1 /store/ariel/flows/payloads/Year/Month/Day/Hour/Minute - Optional. To compare the size of records and payloads without lucene or super indexes, type:
du -hsc /store/ariel/events/{records,payloads}/Year/Month/Day/Hour/ --exclude={super,lucene}
- Event records:
- Determine whether the spike in the stored data matches the spike in the backup size. You must make this comparison four times for flow and event, and payloads and records.
For example, the size of the event records directory triples from 29 July 2022 to 31 July 2022, going from 113M to 361M. This increase in records size indicates an increase in backup size is not an error:361M /store/ariel/events/records/2022/7/31 208M /store/ariel/events/records/2022/7/30 113M /store/ariel/events/records/2022/7/29 682M /store/ariel/events/records/2022/7Note: An increase in a records directory indicates more events or flows came in, while an increase in the payloads indicates the size of their payloads increased whether or not the number of flows or events have. -
As an optional step advanced users, export a backup data archive to an external text file and browse the entries and their corresponding size. Compare a file from before the spike to after the spike to determine what log sources are contributing to the increase in size.
mkdir /store/ibm_support/ mkdir /store/ibm_support/backupinvestigation tar -ztvf /store/backup/backup.nightly.XXX.tgz > /store/ibm_support/backupinvestigation/backup.nightly.XXX.txt
Results
The size of the daily data archive is not always the same each day even if the daily disk usage is similar. If there is no increase in the data stored in Ariel, but the backup variation is too great to be explained by compression differences, contact support.
If there is an increase in the stored data, you can take the following steps to further diagnose this issue:-
Investigate whether log sources were added since the spike.
-
Investigate whether the size of payloads coming from existing log sources increased.
-
Investigate whether any reports were added, as reports are included in the data backup files.
-
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt8AAA","label":"Ariel"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
30 November 2022
UID
ibm16841053