Planning for GPUDirect Storage
IBM Storage Scale support for GPUDirect Storage (GDS) enables a direct path between GPU memory and storage. You need to ensure that certain conditions are met before you start installing the feature.
For more information on GDS, see GPUDirect Storage support for IBM Storage Scale.
The following list provides the prerequisites for installing and using GDS:
- IBM Storage Scale version: Full GDS support including accelerated writes requires IBM Storage Scale 5.1.6 on the GPU client and on the NSD server side. The earlier versions starting with IBM Storage Scale 5.1.2 (IB) or 5.1.3 (RoCE) support accelerated reads and writes in compatibility mode but not accelerated writes. For more details on supported versions, see IBM Storage Scale FAQ in IBM® Documentation. Moreover, if you need to configure GDS for a remote file system, IBM Storage Scale 5.1.2 or later is required on all NSD servers of the remote cluster.
- Data path: GDS requires the NSD path for storage access. The storage must not be locally attached to the GDS clients.
- Storage: ESS and non-ESS storage servers are supported. The storage servers can be in the same or different cluster as the GDS clients.
- GDS requires Mellanox RDMA over InfiniBand or RoCE between GDS clients and storage servers.
- Adapters: Mellanox ConectX-5 or newer adapters are required to use GDS with RoCE.
Supported hardware
The following list provides the hardware requirements:
- GDS clients: x86 with a GPU model that supports GDS. For more details, see NVIDIA GDS documentation.
- Network: EDR or HDR InfiniBand, Ethernet (RoCE).
- InfiniBand adapter: CX5 or CX6. (Mellanox CX4 is OK for IB only if the CX4 firmware is 12.27.4000 or higher.)
Supported software for GDS clients
GDS client must be an IBM Storage Scale NSD client. For information about the supported versions of the required software for GDS clients, see Components required for GDS in IBM Storage Scale FAQ in IBM Documentation.
Supported software for GDS servers
The following list provides the software requirements:
- A storage server for GDS must either be an IBM Storage Scale NSD server or an IBM ESS or ECE.
- Any operating system that supports IBM Storage Scale can be used.
- MOFED: Use the Mellanox OFED stack for the OS.
- For details about driver versions, see Components required for GDS in IBM Storage Scale FAQ in IBM Documentation.