Sizing your hardware for scalable data collection
Before you implement this architecture, consider the hardware requirements for the components that make up your scalable data collection architecture.
The servers can be metal or virtualized. If you use a virtualized server, ensure that hyper threading is disabled and that physical cores are allocated and reserved.
For an example deployment, see Example sizing.
This document outlines the hardware requirements for your scalable data collection components. For more information about the general hardware requirements see the information on Hardware requirements.
The following table summarizes the hardware requirements for each server and the recommended number of servers.
| Component | Daily data ingestion per server in GB | Processors per server | RAM per server in GB | Recommended number of servers |
|---|---|---|---|---|
| IBM® Tivoli® Monitoring Log File Agent | 200 | 8 | 4 | 2 |
| HAProxy | 400 | 8 | 4 | 1 |
| Receiver cluster | 200 | 12 | 32 | 2 |
| Apache Kafka | 200 | 12 | 16 | 2 |
| Sender cluster | 200 | 12 | 32 | 2 |
| Log Analysis server | 400 | 8 | 32 | 1 |
| Indexing Engine server | 400 | 8 | 48 | 1 |
Note the following settings when tuning your hardware:
- For both the Sender and Receiver servers, set the heap size to 16 GB.
- For your Apache Kafka brokers, set the heap size to 4 GB. Set the heap size for Apache ZooKeeper to 1 GB.
- Apache Kafka caches data for 7 days on disk. This means that if you plan to stream 400 GB of data daily, reserve 3.2 Terabytes (TB) of disk space for this cache.
For more information, see Tuning considerations.