Data Nodes and data storage

Data Node information

The following list describes information about Data Nodes:

• Data Nodes add storage and processing capacity.
• Data Nodes are plug-n-play and can be added to a deployment at any time.
• Data Nodes integrate seamlessly with existing deployments.
• Use Data Nodes to reduce the processing load on processor appliances by removing the data storage processing load from the processor.
• Users can scale storage and processing power independently of data collection.
• As of QRadar V.7.2.7, a new data format with native data compression is used. Data is compressed in memory and is written out to disk in a proprietary binary compressed format. The new data format enables a better search performance and a more efficient use of system resources than the previous data format. The previous data format did not have a native built-in compression in older versions of QRadar.

The following diagram shows an example of some uses for Data Nodes in a deployment.

The following list describes the different elements that you need to consider when you deploy Data Nodes.

Data clustering

Data Nodes add storage capacity to a deployment, and also improve performance by distributing data that is collected across multiple storage volumes. When the data is searched, multiple hosts, or a cluster does the search. The cluster can improve search performance, but doesn't require you to add multiple event processors. Data Nodes multiply the storage for each processor.  

Note: You can connect a Data Node to only one processor at a time, but a processor can support multiple Data Nodes.

Deployment considerations

Keep the following information in mind as you set up Data Nodes in a deployment.

• Data Nodes are available with QRadar V7.2.2 and later.
• Data Nodes perform similar search and analytic functions as event and flow processors in a QRadar deployment.
  The operational speed on a cluster is affected by the slowest member of a cluster. Data Node system performance improves if Data Nodes are sized similarly to the Event Processors and Flow Processors in a deployment. To facilitate similar sizing between Data Nodes and event and flow processors, Data Nodes are available on some core appliances. To learn about the specifications of available Data Nodes, select any of the following appliances.

  • QRadar xx05
  • QRadar xx29
  • QRadar xx29-C
  • QRadar xx48
  • QRadar xx48-C
• Data Nodes are available in three formats: software (on your own hardware), physical, and appliances. You can mix the formats in a single cluster.

Bandwidth and latency

Ensure that you have a 1 Gbps link and less than 10 ms latency between hosts in the cluster. Searches that yield many results require more bandwidth.

Appliance compatibility

Data Nodes are compatible with all existing QRadar appliances that have an Event Processor or Flow Processor component, including All-In-One appliances. Data Nodes are not compatible with QRadar Incident Forensics PCAP appliances.

Data Nodes support high availability (HA).

Installation of Data Nodes

Data Nodes use standard TCP/IP networking, and do not require proprietary or specialized interconnect hardware.

Install each Data Node that you want to add to your deployment the same as you would install any other QRadar appliance. Associate Data Nodes with either an event or flow processor. For more information, see Configuring a managed host in the IBM QRadar Administration Guide.

You can attach multiple Data Nodes to a single Event Processor or Flow Processor in a many-to-one configuration.

Decommissioning Data Nodes

Use the System and License Management window to remove Data Nodes from your deployment, as with any other QRadar appliance. Decommissioning does not erase data on the host, nor does it move the data to your other appliances. If you need to retain access to the data that was on the Data Nodes, you must identify a location to move that data to.

Data Rebalancing

Adding a Data Node to a cluster distributes data to each Data Node. If it is possible, data rebalancing tries to maintain the same percentage of available space on each Data Node. New Data Nodes added to a cluster initiate more rebalancing from cluster event and flow processors to achieve efficient disk usage on the newly added Data Node appliances.

Starting with QRadar V7.2.3, data rebalancing is automatic and concurrent with other cluster activity, such as queries and data collection. No downtime is experienced during data rebalancing.

Data Nodes offer no performance improvement in the cluster until data rebalancing is complete. Rebalancing can cause minor performance degradation during search operations, but data collection and processing continue unaffected.

Important: When you add Data Nodes to a cluster, they must either all be encrypted, or all be unencrypted. You cannot add both encrypted and unencrypted Data Nodes to the same cluster.

Note: Encrypted data transmission between Data Nodes and Event Processors is not supported.

Management and Operations

Data Nodes are self-managed and require no regular user intervention to maintain normal operation. QRadar manages activities, such as data backups, high availability, and retention policies, for all hosts, including Data Node appliances.

Data Node failure

If a Data Node fails, the remaining members of the cluster continue to process data.

When the failed Data Node returns to service, data rebalancing can occur to maintain proper data distribution in the cluster, and then normal processing resumes. During the downtime, data on the failed Data Node is unavailable, and I/O errors that occur appear in search results from the log and network activity viewers in the QRadar user interface.

For catastrophic failures that require appliance replacement or the reinstallation of QRadar, decommission Data Nodes from the deployment and replace them using standard installation steps. Copy any data that is not lost in the failure to the new Data Node before you deploy. The rebalancing algorithm accounts for data that exists on a Data Node, and shuffles only data that was collected during the failure.

For Data Nodes deployed with an HA pair, a hardware failure causes a failover, and operations continue to function normally.

SAN Overview

To increase the amount of storage space on your appliance, you can move a portion of your data to an offboard storage device. You can move your /store, /store/ariel, or /store/backup file systems.

Multiple methods are available for adding external storage, including iSCSI, Fiber Channel, and NFS (Network File System). You must use iSCSI or Fiber Channel to store data that is accessible and searchable in the UI, such as the /store/ariel directory, and reserve the use of NFS for data backups only.

Moving the /store file system to an external device might affect QRadar performance.

After migration, all data I/O to the /store file system is no longer done on the local disk. Before you move your QRadar data to an external storage device, you must consider the following information:

• Searches that are marked as saved are also in the /transient directory. If you experience a local disk failure, these searches are not saved.
• A transient partition that exists before you move your data is likely to remain in existence after the move, and it can be mounted on an iSCSI, or Fiber Channel storage mount.

For more information about offboard storage, see the IBM QRadar Security Intelligence Offboard Storage Guide.

Data encryption

Event data in QRadar® is not encrypted when stored, there are a number of things that can be done to protect data that is stored on the system.

• The /store or /store/ariel partitions can be placed on an external device, which uses transparent (to QRadar) cryptography. For more information on external storage options, see Offboard storage overview.
• Event data can be protected from tampering by the use of event hashing. For more information, see Configuring system settings.