The biggest challenge when tuning IBM Spectrum Scale is developing a clear understanding of your goals. Before embarking on a performance tuning exercise it is a good idea to clearly define a performance target.

Developing a performance target

Developing a clear performance target allows you to keep your tuning effort focused. Since tuning is often a game of give and take a focused approach can help you make those value decisions as they arise. A performance target can take many forms:

• A throughput target - " I need to achieve 5 GiB/sec using myapp"
• An IOP target - " I need to be able to create 1,000 files/second using filecreate.bash"
• An application transaction - " I need to complete 100 cube requests a minute"
• A backup or recovery target - " I need to be able to perform a differential backup on 10 million files in 2 hours with a 10% daily change rate and an average file size of 1MB"

Whatever your target there are certain characteristics the goal should have.

• Measurable - the target should be a metric like "files created per second" that can be easily determined.
• Tools - The tools used to execute and measure the test should be clearly defined. You should research your application requirement and find or develop a relevant tool to generate a workload. For the most efficient analysis the results of a test run should be readily apparent and not require extensive post processing before they can be interpreted.
• Relevant - The test workload should be as relevant to the production application as possible. This is the #1 issue with most performance tuning exercises. Many performance tuning engagements fail because the workload is chosen because it is handy, and not necessarily relevant. For example, IOR is a great test of throughput but does not help if you are tuning a file system to provide SMTP email service.

This section contains sample configurations with some discussion around why the specific parameters were chosen. Of course it is always recommended that you test any configuration parameters before implementing them on a production cluster

A general use HPC cluster is typical at many universities. These clusters need to support a variety of I/O workloads so you typically cannot tune to a specific type of I/O pattern. Even with unpredictable IO workloads these clusters have many properties in common. For example, there are usually three types of server usage:

1. IBM Spectrum Scale NSD Servers
2. User login nodes
3. Compute Nodes

These recommendations make many assumptions about server memory and communication infrastructure. Remember these are just examples and are a good place to start though you need to adjust as necessary to fit your environment.

There are slightly different starting points for the three server types

NSD Servers

The assumption here is that these NSD servers are serving the rest of the cluster only. In this configuration you need to account for NSD server IO.

pagepool = 16GB or more - NSD servers do not require large amounts of memory.

maxMBpS = 12000

nsdMaxWorkerThreads=1024

nsdThreadsPerQueue=12

nsdSmallThreadRatio=1

For more details see NSD Server Tuning

User Login Nodes

The assumption here is that these servers are user interactive nodes. The IO throughput requirements may or may not be high but there are many users doing "ls" operations.

pagepool = 16GB

maxMBpS = 12000

maxFilesToCache = 250,000

Compute Nodes

The assumption here is that these servers are used by applications and that memory is best used by the application and not IBM Spectrum Scale cache.

pagepool = 4GiB

maxMBpS = 5000

maxFilesToCache = default

In this configuration you have a small cluster 2-10 nodes that all have direct access to a common set of LUNS over a SAN, for example. All of these nodes are running a variety of applications.

All Nodes

pagepool = 8GB

maxMBpS = 12000

maxFilesToCache = 250,000