Server and rack solutions
Ceph provides both optimized server-level and rack-level solution SKUs. Hardware vendors provide both optimized server-level and rack-level solution SKUs. Validated through joint testing with IBM, these solutions offer predictable price-to-performance ratios for Ceph deployments, with a convenient modular approach to expand Ceph storage for specific workloads.
Many hardware vendors now offer both Ceph-optimized servers and rack-level solutions that are designed for distinct workload profiles. IBM works with multiple storage server vendors to test and evaluate specific cluster options for different cluster sizes and workload profiles. This work is to simplify the hardware selection process and reduce risk for organizations. IBM’s exacting methodology combines performance testing with proven guidance for a broad range of cluster capabilities and sizes.
With appropriate storage servers and rack-level solutions, IBM Storage Ceph can provide storage pools that serve various workloads from throughput-sensitive and cost and capacity-focused workloads to emerging IOPS intensive workloads.
IBM Storage Ready Nodes for IBM Storage Ceph represents an option in the IBM Storage Ceph portfolio. It comprises IBM Storage Ceph software together with a set of IBM Storage Ready Nodes as a unified offering. The Ready Nodes are IBM supported servers that are validated for IBM Storage Ceph. The IBM Storage Ready Nodes for IBM Storage Ceph offering from IBM delivers simplified IBM Storage Ceph deployment and operations, with full stack IBM support and services over the product lifecycle. For more information, see IBM Storage Ready Node for IBM Storage Ceph documentation.
IBM Storage Ready Nodes hardware recommendations for HDD cluster (cost and capacity-optimized)
| Dell PowerEdge R750 X2D SATA | Dell PowerEdge R760 X6D SATA HDDs | |
|---|---|---|
| Processor | Intel Xeon Silver 4314 | Intel Xeon Silver 4314+ 2GHz |
| Number of processors | 2 | 2 |
| RAM | 16x16GB RDIMM | 16x16GB RDIMM |
| OS Disk | 2x M.2 240GB (RAID 1) | 2x M.2 480GB (RAID 1) |
| Data Acceleration Disk | 2x 3.84 TB SSD SATA | 2x 3.84 TB NVMe Read Intensive |
| Rack height | 2U Form Factor | 2U Form Factor |
| Width | 482 mm (18.97 in.) | 482.0 mm (18.98 in.) |
| Depth | 772.11 mm (30.39 in.) | 816.7 mm (32.15 in.) |
| Height | 86.8 mm (3.41 in.) | 86.8 mm (3.42 in.) | 2U |
| Weight | 35.3 kg (77.82 lb) max | 29.55 kg (65.14 lbs) / Fully pop'ed 46.3 kg (102.07 lbs) |
| Capacity disks | 12 | 24 |
| NVMe disk sizes | Not applicable | Not applicable |
| SATA disk sizes | 8TB, 12TB, 16TB, 20TB | 8TB, 16TB, 20TB, 24TB |
| Network ports | 2x1GbE, 2x10GbE | 2x 1GbE Management, 4x 10/25GbE, 2x 100GbE (100 Gb max) |
| Options / upgrades | India power cords x2 | Dual Port 32Gb Fibre Channel HBA, 16GB DDR5 RDIMM 4800MT/s . 2x 3.84TB NVMe Read Intensive |
IBM Storage Ready Nodes Hardware recommendations for SSD cluster (throughput-optimized)
| Dell PowerEdge R760 NVMe X5D | |
|---|---|
| Processor | Intel Xeon Gold 6438N |
|
Number of processors |
2 |
| RAM | 16x16 GB RDIMM |
| OS Disk | 2x M.2 480GB (RAID 1) |
| Data Acceleration Disk | 1x 960 GB NVMeA |
| Rack height | 2U Form Factor |
| Width | 482 mm (18.97 in.) |
| Depth | 772.11 mm (30.39 in.) |
| Height | 86.8 mm (3.41 in.) |
| Weight | 35.3 kg (77.82 lb) max |
| Capacity disks | 24 |
| NVMe disk sizes | 3.84 TB, 7.68 TB, 15.36 TB |
| SATA disk sizes | Not available |
| Network ports | 2x1GbE Management,
2x 1025GbE 2x 100GbE (100 Gb max) |
| Options / upgrades | India power cords x2 |
- Network switching
- Redundant network switching interconnects the cluster and provides access to clients.
- Ceph MON nodes
- The Ceph monitor is a datastore for the health of the entire cluster, and contains the cluster log. Use a minimum of three monitor nodes for a cluster quorum in production.
- Ceph OSD hosts
- Ceph OSD hosts house the storage capacity for the cluster, with one or more OSDs running per individual storage device. OSD hosts are selected and configured differently depending on both workload optimization and the data devices installed: HDDs, SSDs, or NVMe SSDs.
- IBM Storage Ceph
- Many vendors provide a capacity-based subscription for IBM Storage Ceph bundled with both server and rack-level solution SKUs.
IOPS-optimized solutions
With the growing use of flash storage, organizations increasingly host IOPS-intensive workloads on Ceph storage clusters enabling emulation of high-performance public cloud solutions with private cloud storage. These workloads commonly involve structured data from MySQL-, MariaDB-, or PostgreSQL-based applications.
| Element | Specification |
|---|---|
| CPU | 5 physical cores (10 hyperthreads) per NVMe SSD |
| RAM | 24 GB baseline, plus 6 GB per OSD |
| Networking | 10-Gigabit Ethernet (GbE) per 2 OSDs |
| OSD media | High performance enterprise NVMe SSDs, read-intensive or mixed-use |
| OSDs | One OSD per SSD for most deployments |
| BlueStore WAL/DB | High performance enterprise NVMe SSD, colocated on the OSD |
| Controller | Native PCIe bus |
For more information, see Supermicro® storage products.
| Vendor | Small (250 TB) | Medium (1 PB) | Large (2 PB+) |
|---|---|---|---|
| SuperMicro 1 | SSG-122B, SSG-222B, SSG-121, SSG-110P-NTR10 | SSG-122B-NE316R, SSG-110P-NTR10 | SSG-222B-NE3X24R |
Throughput-optimized solutions
Throughput-optimized Ceph solutions are usually centered around semi-structured or unstructured data. Large-block sequential I/O is typical.
| Element | Specification |
|---|---|
| CPU |
|
| RAM | 24 GB baseline, plus 6 GB per OSD |
| Networking | Redundant 25GE or 100GE links |
| OSD media | 7,200 RPM Enterprise HDDs or NVMe SSDs |
| OSDs | One per storage drive |
| BlueStore WAL/DB | Offloaded to shared high-performance NVMe SSDs |
| Controller | RAID HBAs are not recommended; for SAS/SATA drives a JBOD style HBA is preferred. |
Several vendors provide pre-configured server and rack-level solutions for throughput-optimized Ceph workloads. Extensive testing and evaluation of servers from Supermicro and Quanta Cloud Technologies (QCT) has been conducted.
| Vendor | Small (250 TB) | Medium (1 PB) | Large (2 PB+) |
|---|---|---|---|
|
SuperMicro |
SSG-110P-NTR10, SYS-112B-WR, SSG-122B-NE316R | SSG-122B-NE316R | SSG-222B-NE3X24R |
|
QCT1 |
QxStor RCT-200 |
QxStor RCT-400 |
QxStor RCT-400 |
1QxStor Ceph Storage Open Source Edition
| Vendor | Small (250 TB) | Medium (1 PB) | Large (2 PB+) |
|---|---|---|---|
|
Dell |
R670 | R670, R770 | R7715 |
|
Cisco |
C220 M8 | C240 M8 | C245 M8 |
|
Lenovo |
SR645 V3 | SR645 V3 | SR665 V3 |
Cost and capacity-optimized solutions
Cost- and capacity-optimized solutions typically focus on higher capacity, or longer archival scenarios. Data can be either semi-structured or unstructured. Workloads include media archives, big data analytics archives, and machine image backups. Large-block sequential I/O is typical.
| Element | Specification |
|---|---|
| CPU | 2 physical cores (4 hyperthreads) per OSD |
| RAM | 24 GB baseline, plus 5 GB per OSD |
| Networking | Redundant 10 GE, 25 GE, or 100 GE links |
| OSD media | 7,200 RPM Enterprise HDDs or QLC-class SSDs |
| OSDs | One per storage drive |
| BlueStore WAL/DB | Colocated on the storage drive |
| Controller | RAID HBAs are not recommended; for SAS/SATA drives a JBOD style HBA is preferred. |
| Vendor | Small (250 TB) | Medium (1 PB) | Large (2 PB+) |
|---|---|---|---|
|
Dell |
R470 | R570 | R7715 |
|
Cisco |
C225 M8 | C245 M8 | C245M8 |
|
Lenovo |
SR630 V3 | SR645 V3 | SR665 V3 |
- IBM Storage Ceph Concepts and Architecture Guide IBM Redpaper
- IBM Storage Ceph Solutions Guide IBM Redpaper
- Unlocking Data Insights and AI: IBM Storage Ceph as a Data Lakehouse Platform for IBM watsonx.data and Beyond IBM Redbooks
- Deploying MySQL Databases on Red Hat Ceph Storage
- Intel® Data Center Blocks for Cloud – Red Hat OpenStack Platform with Red Hat Ceph Storage