What is disaggregated storage?

Disaggregated storage is a way for organizations to minimize overprovisioning and optimize cost efficiency by allocating resources only as they are needed, while minimizing added latency and managing security by choosing the right network fabric.

Disaggregated storage presents a fundamental change in the relationship between two major technological components: compute resources and data storage. In a traditional coupled system, storage is physically attached directly to compute resources. This setup is called direct-attached storage or DAS. A regular personal computer uses a DAS setup, with a hard disk drive (HDD) or SSD plugged into the same motherboard as the GPU and CPU, as do standard rack servers in data centers. DAS can use serial-attached SCSI (SAS) as the standard to ferry data between storage and compute resources, as well as other interfaces such as SATA and USB. 

In a server rack setup, this coupling presents certain weaknesses. If a user wants to upgrade the storage, it might require the purchase of new servers, which come with their own unneeded compute power—a waste of resources. Alternately, the user might purchase a JBOD unit, which stands for “just a bunch of disks,” and appropriately enough, consists of a basic enclosure containing multiple drives. This functions as a hardwired external storage solution; the host server handles file system organization. But in the future, if that user wants to change, add or upgrade the compute resources, JBODs are limited in sharing with multiple servers and restrained by physical space.

Industry solutions to the limits of DAS run in two opposite directions. On the tighter coupling side sits hyper-converged infrastructure (HCI), which bundles compute and storage together in software so they scale as a unit. Disaggregated storage goes the other direction, separating compute and storage so they can each be scaled individually. 

A storage area network (SAN) might be considered a type of disaggregated storage, in that compute and storage are physically separated and connected over a network. But in a SAN, the storage is controlled with a dedicated, proprietary appliance, whereas newer versions of disaggregated storage rely on cloud software.

Disaggregated storage is an ever-evolving field. Research is even creating new system design variants, such as running RAID on top of disaggregated storage.

In a disaggregated storage model, storage and compute resources are physically decoupled and connected over a network. Software-defined storage (SDS)—a methodology in which a software layer is used to decouple storage resources from an underlying physical storage hardware infrastructure—is used to pool and provision resources. 

SDS policies programmatically govern data placement and I/O paths, and control rates like input/output operations per second (IOPS), or the number of read and write operations performed per second.

Maintaining a high IOPS when using disaggregated storage requires high speeds from both the drive and the network. High-end enterprise SSDs can top a million IOPS, but that speed won’t matter if the network can’t match it. There are a few different network infrastructure varieties and technologies used to get the most out of disaggregated storage:

NVMe-oF is the umbrella term for the network technology that includes the following three subtypes. NVMe itself is a protocol for data transfer designed for flash storage and SSDs. It’s considered to be a replacement for Serial Advanced Technology Attachment (SATA) and SAS standards, which were better used for hard disk drives. NVMe is fast when coupled with a CPU, but when decoupled, traditional network technologies can create a bottleneck. 

NVMe-oF supports up to 64,000 queues, each holding up to 64,000 commands. This parallelism can dramatically boost data access capacity to take advantage of the power of disaggregated storage.    

This is a variety of NVMe-oF that wraps NVMe commands inside standard Transmission Control Protocol (TCP) packets. TCP/IP (the IP stands for internet protocol) is the standard way of connecting computer systems in a network. NVMe-oF over TCP runs on the existing network and does not require new or expensive network hardware. 

However, this method does have slightly higher latency and CPU usage compared to RDMA-based fabrics. Given these attributes, NVMe-oF over TCP is often a good fit for mixed data centers and faster upgrades, and sufficient for enterprise workloads that don’t require the lowest possible latency.

RDMA, which stands for remote direct memory access, is a different way of sending and receiving data than the more typical TCP/IP. Network fabrics that use RDMA require specific RDMA-capable network cards that handle the work of pushing data over the network and pulling it from the network. 

With TCP/IP, the sender and receiver’s CPUs do the heavy lifting of copying data back and forth. With RDMA, the cards do that work by speaking directly to each other, bypassing the CPUs on both sides. It’s a bit like giving a delivery driver keys to your garage, removing the need for you to get up from your desk to accept the package. Instead, the package is just placed in your garage the moment it arrives.

RDMA comes in a few different forms: the most common over standard Ethernet is RDMA over Converged Ethernet, or RoCE, which requires a specially configured lossless network. InfiniBand, which is a separate networking standard rather than a variant of Ethernet, is another common option. It offers lower latency than RoCE or Fibre Channel, and is often used in high-performance computing (HPC) and AI clusters, though it requires organizations to buy and operate a second networking technology.

Because RDMA writes straight into memory and bypasses the receiver’s CPU, it also skips the traffic inspection that the CPU usually performs. RDMA fabrics close this gap by controlling access rather than inspecting traffic. The storage network—the same dedicated fabric that RoCE already requires—is kept separate from general traffic, and the storage system itself controls which servers can access which data.

Fibre Channel is an established technology that still provides the backbone for disaggregated storage in many enterprise SANs. It predates NVMe and traditionally carries SCSI commands; however, FC-NVMe allows NVMe traffic to run on the same fabric.

It is a purpose-built storage fabric that natively provides the lossless behavior that RoCE must be configured for. However, as with InfiniBand, it requires a second networking technology, which is one reason why new deployments often favor Ethernet-based fabrics.

Network file system (NFS) and server message block (SMB) are file-level protocols that network attached storage (NAS) devices use to serve files. In a technical sense, maintaining an NAS, whether that’s a physical box or certain cloud file storage options, could be considered disaggregated storage. After all, the storage is separated from the compute resources and either can be altered without altering the other.

However, NFS and SMB come with higher latency, heavier CPU usage and concurrency bottlenecks compared with block protocols like NVMe-oF. And while capable of serving disaggregated storage, NFS and SMB require specific high-performance variants such as Server Message Block Direct of NFS over RDMA to be suitable for HPC workloads like training AI models or high-traffic database queries.

Replication is exactly what it sounds like: a process that copies data to multiple locations for redundancy. It requires storage equal to the size of the data multiplied by the number of replicas. It can tolerate as many simultaneous failures as there are replicas, minus one, since there must be one intact copy left.

Erasure coding is more storage-efficient. It splits data into many parts and stores them in many different locations. The storage system creates one or more parity fragments—mathematically computed redundancies that enable the system to rebuild lost data. The number of parity fragments determines how many failures the system can survive.

Erasure coding typically needs about 1.5 times the raw capacity of the data to tolerate two simultaneous failures. Replication would require 3 times the raw capacity for the same fault tolerance.

The appeal of disaggregated storage comes from several key benefits:

In tightly coupled architectures, it can be difficult, limiting or expensive to scale either storage or compute power separately, rather than both at the same rate. For example, an AI company might suddenly acquire 100 terabytes of new training data but not need any additional compute power. A disaggregated system enables the organization to add storage without overprovisioning compute.

Compute hardware generally turns over faster than storage, since new generations of CPUs and GPUs often offer performance gains that justify the upgrade, while storage upgrades arrive more slowly and matter less to many workloads.

In a tightly coupled system, organizations either need to discard storage that still has plenty of life left to upgrade compute, or stick with compute that they would prefer to replace just to get the most out of storage resources. Disaggregation lets an organization install new compute nodes while storage stays in place.

A disaggregated system enables multiple servers to read the same data at the same time, which can be extremely useful in scenarios where multiple departments need to work off the same information in real time, such as the reconciliation of financial data. Without shared access, each server needs its own copy of data, and those copies must be kept in sync. Shared access presents a single source of truth, reducing the potential risks of data siloing. 

Because compute and data are separated and connected over a network, when a compute node fails, it doesn’t take data with it. System maintenance no longer takes storage offline; workloads can move between servers during updates without the data moving. 

The tradeoff is that this scenario introduces a new dependency: reaching the data requires a healthy network. 

While disaggregated storage provides significant advantages, it’s not without downsides. The cost, network dependencies and failure scenarios can combine to make disaggregated storage an inappropriate solution in certain situations.

Disaggregated storage relies on a network to connect storage and compute. In such a system, the network must also be designed for redundancy and consistently monitored so that congestion or misconfigurations do not sever healthy servers from healthy data. A shared network also introduces the “noisy neighbor” effect, in which one workload can influence the latency of another.

Disaggregated storage introduces new failure modes, such as the “split brain” issue. This is when nodes lose contact with each other, resulting in multiple nodes writing data concurrently without a way to reconcile the multiple versions once a connection is restored. This risk can be mitigated through quorum and fencing—requiring that a majority or nodes agree before accepting writes and cutting off nodes that fall out of contact; however, it’s an additional complexity to contend with. 

Setting up a disaggregated system can require significant upfront costs. Specialized hardware is essential for RoCE, including RDMA-enabled network cards, switches configured for lossless operation and high-speed cabling. Even using NVMe-oF over TCP often involves software licensing and a higher upfront cost for separate servers and storage, though that cost difference might decrease over time with more upgrades and additions.

For organizations with predictable workloads where compute and storage scale at the same rate and shared access to the same data is less crucial, DAS often remains a suitable choice.

The benefits of disaggregated storage make it a common choice for certain key use cases:

Training AI models requires massive amounts of data storage and compute, and these two resources might not always scale in lockstep. Disaggregated storage lets operators add storage when necessary to accommodate huge sets of training data, and to add GPUs only as necessary for processing—especially valuable given the high demand and cost of GPUs.

Idle GPUs awaiting data are expensive, and training often runs across many nodes reading the same dataset. A disaggregated system creates a shared storage pool on a high-speed network that can feed many GPUs at once from the same dataset, without the need to copy data onto every server.

Distributed databases spread their data storage and compute power among multiple servers. Disaggregated storage enables any node to serve any query, since every compute node can reach the entire dataset rather than just what’s stored locally.  

Hyperscale cloud providers rely on disaggregated storage to keep their operations efficient and current. CPU and GPU technology moves quickly, and disaggregation enables these providers to remove outdated compute hardware and replace it with more modern hardware without losing or migrating any customer data. 

In addition, disaggregation optimizes costs by allocating only the storage and computing power a given customer needs at any given point.