Question & Answer
Question
How can I reduce the amount of storage used?
Answer
The following are items that can be taken into consideration when attempting to reduce the amount of storage used:
As data is collected on QRadar, it is stored on disk. If you can enable the option on your retention requirements, to delete data that is outside your retention window.
Using multiple Retention Buckets, you can partition, or divide data into multiple containers. These containers can then be configured with specific retention, deletion options. For example, if you have a number of log sources that generate numerous data, but you need only to keep them for short periods of time, you could put them into a retention bucket. Use a force-delete option of the time you need for that data. Proxy logs are examples of this, that some users might decide to force-delete after a period of 1 week, allowing other data to remain on disk for a longer time period.
Conversely, you might want to keep a particular set of data for a longer time period, such as compliance systems, for up to a year. If the rest of your data could be deleted after a month, allowing more space to keep that longer term data, thus again, by using less space overall.
Data on disk is indexed across multiple properties, approximately 15, which are enabled by default. If you are not using those indexes often for searches, you might want to disable them in Index Management, to reduce both disk space utilization, and IO cycles used to write them as data is collected.
The Quick Search Index is a full text index of all data received by QRadar, and is kept for 30 days by default. If during your use of this index, you are often limited to searching only the last weeks worth of information, you might want to reduce the retention of this index. The size depends on the variety of data, but on average, increases usage by about 40% for the time period kept. For example, if your data collected in a day was 100GB in size (raw records), the Quick Search Index for that would be approximately 40GB per day, in addition to the 100GB collected.
Forced Deletion of Data Outside Retention Requirements:
As data is collected on QRadar, it is stored on disk. If you can enable the option on your retention requirements, to delete data that is outside your retention window.
Use Retention Buckets to Partition Data:
Using multiple Retention Buckets, you can partition, or divide data into multiple containers. These containers can then be configured with specific retention, deletion options. For example, if you have a number of log sources that generate numerous data, but you need only to keep them for short periods of time, you could put them into a retention bucket. Use a force-delete option of the time you need for that data. Proxy logs are examples of this, that some users might decide to force-delete after a period of 1 week, allowing other data to remain on disk for a longer time period.
Conversely, you might want to keep a particular set of data for a longer time period, such as compliance systems, for up to a year. If the rest of your data could be deleted after a month, allowing more space to keep that longer term data, thus again, by using less space overall.
Disable Unused Indexes:
Data on disk is indexed across multiple properties, approximately 15, which are enabled by default. If you are not using those indexes often for searches, you might want to disable them in Index Management, to reduce both disk space utilization, and IO cycles used to write them as data is collected.
Adjust Quick Search Index:
The Quick Search Index is a full text index of all data received by QRadar, and is kept for 30 days by default. If during your use of this index, you are often limited to searching only the last weeks worth of information, you might want to reduce the retention of this index. The size depends on the variety of data, but on average, increases usage by about 40% for the time period kept. For example, if your data collected in a day was 100GB in size (raw records), the Quick Search Index for that would be approximately 40GB per day, in addition to the 100GB collected.
Related Information:
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.3;7.5.0"}]
Was this topic helpful?
Document Information
Modified date:
02 March 2023
UID
swg21993401