Evaluation of Spectrum Scale Performance on the Spectrum Symphony Offering on IBM Cloud

6 min read

Using IBM Spectrum Scale and IBM Spectrum Symphony on IBM Cloud VPC.

The use of IBM Spectrum Scale storage for high performance computing (HPC) workloads is now available as part of the IBM Spectrum Symphony offering on IBM Cloud. Together, Spectrum Scale and Spectrum Symphony offer an attractive platform for compute- and storage-intensive tasks like financial risk-analysis workloads.

A related offering is planned soon that will allow users to create stand-alone Spectrum Scale storage clusters that can be used with just about any type of compute workload. The following diagram shows the layout of a combined Spectrum Symphony compute cluster and the Spectrum Scale storage cluster created by this offering. Note that the compute and storage clusters are separate, which would allow compute nodes to be deactivated for cost savings while the storage nodes can be left active to retain the stored data:

Figure 1. Spectrum Symphony and Spectrum Scale offering cluster layouts.

Figure 1. Spectrum Symphony and Spectrum Scale offering cluster layouts.

Instance storage in IBM Cloud VPC

The current IBM Spectrum Scale offering is based on VPC instance storage. You can find an introduction to the basic functional and performance characteristics of the technology in this blog post: “Understanding Instance Storage on IBM Cloud Virtual Servers for VPC.” Instance storage is provided by one or more solid state drives residing in a VPC physical server. This storage is then made available to virtual server instances to provide high-performance ephemeral storage for any purpose.

In the case of the evaluation described in this blog, the purpose is to create nodes in a Spectrum Scale storage cluster. To see the available instance storage VSIs using the IBM Cloud command line interface (CLI), run the following command: ibmcloud is instance-profiles | grep x2d.  You will see profiles with a variety of VCPU, memory and storage options. 

Creating a storage cluster

The large number of instance storage VSI profiles available — each with different storage and compute resources — when combined with the ability of Spectrum Scale to use up to 18 storage nodes (the maximum currently supported by our Spectrum Symphony offering), presents a challenge when planning a storage cluster. While carefully designed automation eases the task of provisioning, configuring and mounting the storage, the number and type of storage and compute instances is up to you. This blog post is designed to help you build a storage cluster that meets your needs.

Considerations when designing a Spectrum Scale storage cluster

Four characteristics will likely guide your choices for a storage cluster

  1. Capacity: How much storage do you need?
  2. Read and/or write bandwidth: Will data transfer speeds have a significant effect on your workload performance?
  3. IOPS: For many workloads, the IOPS rate of even the smallest storage clusters will likely prove sufficient, but if your workload is IOPS hungry, your cluster can be built to deliver 1 million or more IOPS.
  4. CPU capacity: Sustaining a high rate of I/O operations in a storage cluster also taxes the CPU resources. In general, the instance storage configurations that IBM Cloud provides are sized to provide sufficient CPU resource while operating at maximum IOPS.

Looking at the performance of some sample clusters

As part of performance testing, we created and tested several different cluster configurations and capacities. Let’s look at some of the data from this work and then continue the discussion of how you might want to construct your storage cluster:

Table 1. Storage cluster configurations and their IOPS and bandwidth capabilities.

Table 1. Storage cluster configurations and their IOPS and bandwidth capabilities.

To give some context to these numbers, it may be useful to look at the performance characteristics of the raw volumes provided to an instance storage VSI. Here are those numbers:

Table 2. Instance storage disks and their measured performance characteristics.

Table 2. Instance storage disks and their measured performance characteristics.

The instance storage is presented to the VSI as a virtual disk(s). Each disk is created using a portion of the storage provided by an NVMe device owned by the instance server. Since the NVMe device storage is divided among virtual instances using namespaces, the bandwidth and IOPS are controlled to preserve consistent QOS for all users. The numbers shown above were obtained using the fio benchmark and should match what you would see on your VSI if you ran the same tests.

Looking at the numbers, a few things stand out. Note the read and write bandwidth scale with volume size on the raw disks — for the Spectrum Scale clusters, the bandwidth available is essentially the total bandwidth of all the disks added together. For the single disk instances, IOPS scale is proportional to capacity, but on two disk instances, the total IOPS available is somewhat less than proportional to disk capacity.  Based on these observations, here are a couple of recommendations:

  • If you have an IOPS-hungry workload, you may want to build your storage cluster with single disk instances to maximize IOPS. 
  • If you are more concerned with bandwidth, you can reduce the complexity of your cluster without sacrificing bandwidth by choosing two disk nodes like the cx2d-32x64.

Network bandwidth considerations

The IBM Spectrum Scale filesystem is a distributed filesystem where the nodes are connected by a network. This means that in addition to storage performance, the overall performance of the storage cluster is also dependent on consistent network performance and subject to network bandwidth limitations. 

The IBM Cloud network is both reliable and consistent, and the current bandwidth limit for a single vNIC is 25 Mb/s. While multiple vNICs are allowed to be configured per VSI, we deploy the storage nodes each with only a single vNIC. It should be noted that there is also a system-level aggregate bandwidth limit per VSI that is proportional to the number of VCPUs. This comes into play for VSIs with less than 16 VCPUs. For additional details about instance bandwidth, see the “Bandwidth Allocation in Virtual Server Instances” blog.

Another factor in reaching maximum throughput for the storage cluster is the number of compute nodes and their network links. Converting 25Mb/s to bytes gives approximately 3GB/s, but looking at Table 1, it is apparent that the total bandwidth of the storage cluster can easily exceed the bandwidth on a single network link. This shows that we must consider the number of links from compute nodes because it takes multiple compute nodes to drive max bandwidth or IOPS. Figure 2 illustrates this for bandwidth. In the case of the 4.8TB eight-node cluster, it takes a minimum of four compute nodes to drive maximum bandwidth:

Figure 2. IBM Spectrum Symphony nodes required to drive maximum bandwidth.

Figure 2. IBM Spectrum Symphony nodes required to drive maximum bandwidth.

CPU requirements of storage clusters

The 4KB random read workload driving max IOPS is a good test for simulating the maximum CPU demands of a storage cluster. The following table adds some CPU utilization metrics to the IOPS numbers from our overview table above:

Table 3. Storage node CPU utilization during IOPS intensive workload runs.

Table 3. Storage node CPU utilization during IOPS intensive workload runs.

The CPU util column lists the average CPU utilization of the storage nodes in the cluster. The results show that running a Spectrum Scale storage cluster at maximum IOPS is a fairly CPU-intensive exercise. This is unsurprising given the demands of delivering hundreds of thousands of IOs per second.  

Another observation is that there is some loss of CPU efficiency per IOP as the number of nodes in the cluster is increased. This can be seen in the IOPS per VCPU numbers. In general, however, all the clusters we tested had sufficient CPU capacity to handle maximum IOPS.

More on instance profiles

If you used the CLI to list the instance profiles, you probably noticed that there are quite a few of them, and only a small number are discussed in this blog post. The profiles we have chosen are all fairly small and from the compute family. In general, the compute family of instances are a good fit for a Spectrum Scale cluster because the other types offer more memory that is not needed for these storage clusters. As you can see, there are also a few larger nodes that offer more storage, which we did not include in this discussion. If you need more than the 20TB of storage offered by 18 cx2d-32x64 instances, the larger instances are available. For 10TB or smaller, the 8-32 VCPU instances tend to be a good fit for value and performance.

Putting it all together

One of the many great attributes of the cloud is the fact that you are never locked into specific hardware. If you choose a storage cluster and later need more capacity or performance, it can always be remade with larger resources. We hope that the information provided in this blog post will help you get started in creating IBM Spectrum Scale storage for your IBM Spectrum Symphony clusters that exceeds your expectations.

Learn more about IBM Spectrum Scale.

Learn more about IBM Spectrum Symphony.

Be the first to hear about news, product updates, and innovation from IBM Cloud