Performance considerations

While default configurations on the IBM Storage Scale and cloud services work for the migration and recall of data, some adjustments might be required for optimized performance. This topic addresses some of such adjustments that will likely have the most significant effect on performance.

IBM Storage Scale cluster hardware

Memory, CPU, storage, and network play a significant role in determining how well the cloud tiering is going to perform.

Memory: cloud services requires at least 2 GB RAM for the heap memory, and the IBM Storage Scale page pool should also be reasonably high. It is recommended to have a minimum of 64 GB RAM for page pool on the Gateway nodes.

CPU: cloud services can run more threads to transfer data if the system has more CPU cores available. For better performance, it is recommended to have a minimum of 24 or 32 CPU cores (the more, the better) on the cloud services gateway nodes.

Storage: The read/write performance from the IBM Storage Scale file system is one of the major factors that determine the migrate/recall performance. To achieve desirable performance, the read/write throughput should be more than the available network bandwidth to the cloud. Some of the general recommendations are:
  • Have flash disks for storing metadata on the file system.
  • If possible, have a separate flash-based file system to keep the cloud services database. This will ensure that the internal cloud service operations that can span the entire database (like reconcile) run in a reasonable amount of time.
    Note: Adding flash storage helps particularly with scaling up the number of files that are supported by cloud services and is apparent when you get to the hundreds of millions of files migrated to the cloud. So, this recommendation on flash storage only applies if you are scaling to those kinds of numbers.

Network: It is recommended to use 10 GB Ethernet for the network interfaces that are used to transmit data to the cloud. In case of on-premise object store, the object store also should have 10 GB Ethernet. It is recommended to have a dedicated external links to the cloud storage and not to use these interfaces for intra-cluster communication.

Software: cloud services uses the Data Management Application programming Interface (DMAPI) of the IBM Storage Scale to read and write the data from the file system. As cloud services can migrate data only as fast as it might read from the file system, it is important to ensure that the read/write performance of the file system is good. Set the following key configuration parameters by using the mmchconfig command:
  • pagepool – 16 GB
  • maxGeneralThreads – 1280
  • dmapiWorkerThreads – 64
  • maxFilesToCache – 100k
  • workerThreads – 1024
Another important IBM Storage Scale configuration is the file system block size. It is recommended to have the block size greater than the cloud services slice size (default 512 KB). This will ensure that you do not read from the disk more than once to fill in a single slice.

cloud services configurations

cloud services has configurable parameters, which can be adjusted based on the environment:
  • connector.server.migrate-threadpool-size – This determines the number of internal threads that accept the migrate requests from the command line. The default value is 32, and is ideal for most of the situations where you have up to 32 CPU cores. If you have a gateway node with more than 32 CPU cores, it is advisable to increase the number to match the number of CPU cores that you have, so cloud services can process more migrate requests at once.
    Note: This recommendation is for Intel servers. For Power® LE servers, you can efficiently run two threads per CPU core, so adjust your thread count accordingly.
  • connector.server.recall-threadpool-size – Normally you do not adjust this parameter. This determines the number of internal threads that accept recall requests from the command line. If you are sure of your workload, you can adjust the migrate and recall thread pool size. For example, if you have a very cold archive application and expect more migrates and a very limited number of recalls, then you could increase the migrate thread pool size and reduce the recall thread pool size.
For more information, see Tuning cloud services parameters.

Recommendations on number of Gateway nodes: While increasing the number of gateway nodes to migrate data can increase the total performance, remember that you must find an optimum limit. For example, if the network link from each node to the cloud is 10 Gb and overall read throughput of the file system is 2 GB/s on cluster, just two gateway nodes could achieve the maximum throughput possible. Depending on the file system read/write throughput and the network link, you must make an informed decision on what should be the optimal number of gateway nodes.

Recommendation on number of Cloud Storage Access Points (CSAPs): If you are going to a public cloud service such as the public IBM Cloud® Object Storage (COS) service, there is no need to add in multiple access points since these services have a built-in load balancer. For on-premise object storage, you may choose to add in multiple Cloud Storage Access Points (CSAP) if you are not using a load balancer as follows:
  • If there are multiple gateway nodes in the cluster, using more than one CSAP for a Cloud Account is a good idea. The number of requests that can be processed by a single Cloud Storage Access Point is limited, and multiple nodes hitting the same access point can cause the performance to drop. A general guideline is to have about as many CSAPs as the number of gateway nodes.

Mixing migrate and recall workloads

cloud services internally shares threads that carry out an operation on each slice of data, as well as threads that carry out the interactions with cloud object storage. Hence, running a huge workload of migrates and recalls simultaneously, would affect the total turn-around of each activity, compared to running migrates and recalls separately. cloud services expects migrates to work in batches, where policy identifies a set of cold files to be moved to a cloud storage tier. Recalls, however, are expected to be accessed on demand and are given higher priority than migrates. It is wise to schedule migration policies where possible during off-peak hours to avoid recall performance impacts.
Note: Bulk policy-based recall operations are supported, but not expected to be the common method for recalling files. Since recalls are generally expected to be on demand, recalls are prioritized ahead of migrates.

Distributing files using policies

IBM Storage Scale policies can be used to distribute the migrations and recalls to all the nodes in the node class. To effectively use the policy, refer to the following points:
  • The mmapplypolicy takes two parameters for the number of parallel threads processing the files (-m) and number of files processed per thread (-b).
    • –m has a default value of 24. If you have more CPU cores available, it is advisable to adjust the –m value accordingly
    • –b has a default value of 100. You can increase the number if you want to process more files in one iteration.
    • Plan the –m and –b carefully, so that all the nodes are given equal number of files. For example, if you have 5000 files to migrate and 4 nodes with cloud services installed, the default values of –m 24 and –b 100 will try to distribute 2400 files to each node. There will not be enough files to be distributed to all nodes, and two nodes will be idle.
  • -N parameter of the mmapplypolicy will accept the node class name, so the policy will distribute the files to all the nodes in the node class.

Using multiple file system or file sets for the same node group

A customer can configure multiple file systems and multiple file sets for a single node group, and run migrates in parallel for all of them. If they are looking for high performance, it is recommended that they configure the policy in a manner that, migrates run on a single file system or file set at a time. cloud services design uses a single server process, and resources such as memory, networking and threading are shared between the containers. Running parallel migrates can split the resource, and hence the effective throughput of a single container can be impacted.

Migration policy runs should not be done during the weekly and daily maintenance windows as migrations conflict with some activities that lock the Cloud directory for long periods of time.

Container spillover

For achieving better performance and ease of maintenance, it is recommended to keep no more than 100 million entries per container. After that, the customer must create a new container for the same file system/fileset and continue migrations. This will ensure that a new database is created for the new container, and the container cloud database size does not grow too large. Operation such as reconcile for a 100 million file database can take a few hours. Growing the size much beyond that results in maintenance operations taking too long since the duration of these operations relative to database size is non-linear.

Note:

The automatic container spillover (auto-spillover) is enabled by default during container creation. Auto-spillover is applicable for containers used with tiering service only. Auto-spillover is not applicable for containers used with sharing service. After reaching the default 100 million entries in a container, as an administrator you need not manually create the new container for the same file system anymore. When number of entries in an Active container reaches the given threshold value, a new spillover container is automatically created by the Cloud service during the next reconcile operation on container. This container is made an Active container for the configured file system or fileset, for subsequent data migrations. An index value is used as a suffix for new container name. For example, when a container named testContainer reaches spillover threshold value, a new container named testContainer001 is created automatically by the Cloud service. The next spillover index can be seen in the output of mmcloudgateway containerpairset list.

Container level auto-spillover settings can be tuned using mmcloudgateway containerpairset create / update. More details of auto-spillover CLI options are available under mmcloudgateway help. You can create a new container for the given file system at any time, even before reaching configured threshold entries per container. Auto-spillover occurs only when the Cloud service finds an active container crossing the configured threshold entries.

Object storage configurations

Use of a load balancer

Object stores such as IBM Cloud Object Storage (formerly known as Cleversafe®) gives you an option to have more than one node that can accept the requests from the client. While using a single CSAP, cloud services can send the requests to only one cloud end-point URL. You can overcome this restriction by using a load balancer to distribute the requests to multiple endpoints.
  • NGINX(https://www.nginx.com/) - IBM Cloud Object Storage recommends the use of NGINX with COS, and has a solution guide, which can be used to configure NGINX with COS. For most customers, this load balancer does not perform as well as the native load balancer provided by cloud services.
  • HAProxy (http://www.haproxy.org/) can be configured on a system, and used to send the requests to more than one end point
  • You can use a simple DNS-based load balancer where each DNS lookup for a host name returns an IP address for a different end-point. The host name can be used to create an account with the cloud services.

If you would prefer that cloud services distribute the requests to multiple cloud storage endpoint URLS, you can configure more than one CSAP, and cloud services uses a very basic load balancing algorithm to distribute load across those CSAPs.

IBM Cloud Object Storage

For on-premise configurations, it is recommended to have multiple accessors setup, so that a load balancer or cloud services itself will have no single point of failure in its cloud services with COS. For public IBM® COS service, there is a built-in load balancer – no additional CSAP or load balancer work is necessary. For public IBM COS service, there is a built-in load balancer, so no additional CSAP or load balancer work is necessary.

Amazon S3

Use the appropriate supported Amazon S3 region while configuring the cloud account. Load balancing or multiple CSAPs are not necessary as Amazon S3 has a built-in load balancer.