How To
Summary
This technote provides information about the ariel_offline_indexer.sh script. This script is used within QRadar to remap information related to events stored in ariel nonstructured database in case of migration, reallocation of events, and more.
Objective
Executing this script (ariel_offline_indexer.sh) re-creates your indexes from the already parsed and written data inside the /store/ariel database.
When is this script needed? Several reasons can be listed matching this scenario, from which we can list:
- Migration of events from different environments. For more information, see QRadar SIEM Hardware Migration Scenarios or Migrating ariel event and flow data between QRadar appliances.
- Re-adding hosts to the deployment with different IPs.
- Corruption of super indexes inside /store/ariel database causes the searches to deliver 0 results. This symptom is the most common when you get no results inside Log Activity tab at a certain time frame.
Environment
The ariel offline indexer script is designed to reindex indices used in searches, dashboards, and reports to provide faster results on these activities. The script is located in the following location /opt/qradar/bin directory.
You can use the -h flag to see a list of options available for the script.
/opt/qradar/bin/ariel_offline_indexer.sh -h
usage: options
-R,--repair re-build corrupted super indices
-d,--duration time duration to look files for in minutes, for
example -d 5
-n,--name ariel data base name, for example -n events
-t,--endtime end time, for example -t "2023/03/20 19:18",
optional, by default current system time
-F,--renamefrom rename from (internal use)
-L,--light load minimal QRadar frameworks
-T,--renameto rename to (internal use)
-V,--validate validate super indices
-a,--auto backfill all active indexes
-b,--batchmode run in batch mode with options in a file
-f,--fts create free text search indices
-h,--help print this message
-k,--key property java class name
-l,--list list all enabled indices from the configuration
-p,--param optional parameter for property (key creator
construction)
-r,--remove remove indices for a property
-s,--superindices create super indices from the minute indices
-v,--verbose verbose (optional, default = false)
-w,--threads maximum number of threads to produce minute indices
if requested, default is 8
For more information about the types of managed hosts and their specific functionality, see QRadar component types.
The ariel_offline_indexer.sh script reindexes data on the local host when executing. If data on multiple hosts requires reindexing, you must run the script on each host separately.
Steps
In this example, we show how to reindex 2 years worth of events located in the following directories that do not appear in the Log Activity pane:
/store/ariel/events/records/yyyy/mm/dd/
/store/ariel/events/payloads/yyyy/mm/dd/
- SSH on to the QRadar console.
- To verify the data, use the du utility to inspect the amount of space used from different years, months, days:
du -xchd 1 /store/ariel/events/records/ du -xhcd 1 /store/ariel/events/payloads/ - In order to get the whole 2 years reindexed, we first have to figure out the complete time frame in which they were written to ariel. If we confirmed that the last event was written at 2022 17 July 5 PM, we can use the script with these arguments:
/opt/qradar/bin/ariel_offline_indexer.sh -n events -t "2022/07/17 17:00" -a -s -d 1051200In this last example, we requested the script to affect the database of events (it can reindex flows as well), starting at 2022 17 July 5 PM and going back 1051200 minutes (which are precisely 2 years). -
If the events we need to reindex are not that far back in time and, for example, were lastly written on the current date we would need to execute the script like this:
/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -s -d 1051200Since the argument -t is not provided, the script assumes it starts from today date backwards 1051200 minutes (2 years).
Result
When the script finishes, it provides a summary of its activity. Example output:Mar 20, 2023 3:20:57 PM com.mchange.v2.log.MLog <clinit> INFO: MLog clients using java 1.4+ standard logging. Mar 20, 2023 3:20:57 PM com.mchange.v2.c3p0.C3P0Registry banner INFO: Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10] Mar 20, 2023 3:20:58 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> com.q1labs.frameworks.core.FrameworksC3P0ConnectionDebugger, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2k2sy6av4o5ag1mj6962|74a4b8fd, debugUnreturnedConnectionStackTraces -> true, description -> null, driverClass -> org.postgresql.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2k2sy6av4o5ag1mj6962|74a4b8fd, idleConnectionTestPeriod -> 300, initialPoolSize -> 5, jdbcUrl -> jdbc:postgresql://127.0.0.1/qradar, maxAdministrativeTaskTime -> 60, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 15, maxStatements -> 0, maxStatementsPerConnection -> 50, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 1200, usesTraditionalReflectiveProxies -> false ] Creating /store/ariel/events/records/2023/3/20/15/super/DestinationPort~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/HasIdentity~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/DeviceId~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/DeviceType~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/Category~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/DestinationIP~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/UserName~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/CREEventList~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/PartialMatchList~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/SourceIP~0 - done Creating /store/ariel/events/records/2023/3/20/15/super/Qid~0 - done ...
Additional Information
Using this utility can take a long amount of time, so it is suggested to run it inside a screen session so the script is not interrupted if connection to the terminal session is lost or dropped.
Remember you can tune your processes inside Red Hat Enterprise Linux (RHEL) and you can assign it a new nice number to get more resources allocated once the PID is identified by using this tool:
ps -ef | grep -i offline_indexer
renice -n -20 -p PID
Another useful aspect is you can pass the arguments with 'batch mode' (-b). You need to create a file with the specific list of arguments, each execution separated by line and you can run it adding the -b and the name of the configuration file. Again, it lists the progress and conclusion, but it deletes the configuration file when it finishes executing.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt8AAA","label":"Ariel"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Product Synonym
SIEM QRadar
Was this topic helpful?
Document Information
Modified date:
06 June 2023
UID
ibm16964604