Creating reference data collections by using the command line

Use the command line to manage reference data collections that cannot be managed in IBM QRadar, such as reference maps, map of sets, map of maps, and tables. Although it's easier to manage reference sets using QRadar, use the command line when you want to schedule management tasks.

About this task

Use the ReferenceDataUtil.sh script to manage reference sets and other types of reference data collections.

When you use an external file to populate the reference data collection, the first non-comment line in the file identifies the column names in the reference data collection. Each line after that is a data record that gets added to the collection. While the data type for the reference collection values is specified when the collection is created, each key is an alphanumeric string.

The following table shows examples of how to format data in an external file that is to be used for populating reference maps.

Table 1. Formatting data in an external file to be used for populating reference data collections
Type of reference collection	Data formatting examples
Reference map	`key1,data` `key1,value1` `key2,value2`
Reference map of sets	`key1,data` `key1,value1` `key1,value2`
Reference map of maps	`key1,key2,data` `map1,key1,value1` `map1,key2,value2`

You can also create reference data collections by using the /reference_data endpoint in the QRadar RESTful API.

Procedure

Using SSH, log in to IBM QRadar as the root user.
Go to the /opt/qradar/bin directory.

To create the reference data collection, type the following command:

./ReferenceDataUtil.sh create name 
[SET | MAP | MAPOFSETS | MAPOFMAPS | REFTABLE] 
[ALN | NUM | IP | PORT | ALNIC | DATE] 
[-timeoutType=[FIRST_SEEN | LAST_SEEN]] [-timeToLive=]

To populate the map with data from an external file, type the following command:
```
./ReferenceDataUtil.sh load name filename
 [-encoding=...] [-sdf=" ... "]
```

Example

Here are some examples of how to use the command line to create different types of reference data collections:

Create an alphanumeric map:

./ReferenceDataUtil.sh create testALN MAP ALN

Create a map of sets that contains port values that will age out 3 hours after they were last seen:

./ReferenceDataUtil.sh create testPORT MAPOFSETS PORT 
-timeoutType=LAST_SEEN -timeToLive='3 hours'

Create a map of maps that contains numeric values that will age out 3 hours 15 minutes after they were first seen:
```
./ReferenceDataUtil.sh create testNUM MAPOFMAPS 
NUM -timeoutType=FIRST_SEEN -timeToLive='3 hours 15 minutes'
```

Create a reference table where the default format is alphanumeric:

./ReferenceDataUtil.sh create testTable REFTABLE 
ALN -keyType=ipKey:IP,portKey:PORT,numKey:NUM,dateKey:DATE

What to do next

Log in to QRadar to create rules that add data to your reference data collections. You can also create rule tests that detect activity from elements that are in your reference data collection.