Sending data to the server with the object agent service

Edit online

When you send data to the IBM Storage Protect server with the object agent service, performance generally improves in relation to the size of the input/output operation.

Guidelines for optimal performance

The following circumstances help to improve performance:
  • Client objects are stored by using a single HTTP PUT operation.
  • A multipart upload process is used as part of the S3 protocol. A client object is separated into multiple parts that are transferred by using individual HTTP PUT operations, which can occur serially or in parallel.

Table 1 lists the guidelines for optimal performance when you send data to the IBM Storage Protect server by using the object agent service.

Table 1. Guidelines for sending data to the server with the object agent service
Network type Guidelines for optimal performance

Low-latency networks with low or no data loss, such as a local area network (LAN)
Client objects of less than 100 MB

For client objects that are less than 100 MB, send the client object to the server with a single HTTP PUT operation.

Client objects of 100 MB or greater

For client objects that are 100 MB or greater, use the S3 protocol multipart upload service to send the client object to the server.

Higher-latency networks, such as a wide area network (WAN), for example, the internet
Client objects of 50 MB

For client objects that are less than 50 MB, store the client object with a single HTTP PUT operation.

Client objects of 50 MB or greater

For client objects that are greater than 50 MB, use the S3 protocol multipart upload service to send the client object to the server.

For networks where data loss is a concern, use the minimum part size of 5 MB to avoid retransmitting large amounts of data. Retransmission of parts might be necessary for the client application if data loss occurs.

Guidelines for multipart upload

When you use the S3 protocol multipart upload service to send the client object to the server, consider the following service characteristics:
  • The minimum part size for multipart upload with the S3 protocol is 5 MB.
  • Throughput performance generally increases with larger part sizes up to approximately 50 MB.
Table 2 lists guidelines and examples for sending data to the server with the S3 protocol multipart upload service.
Table 2. Guidelines for sending data to the server with the S3 protocol multipart upload service
Upload characteristics Guidelines for optimal performance Examples for multipart upload
  • Network data loss is not a concern.
  • Multipart upload is done serially.
  • A single client application thread sends the parts of a client object sequentially.

Use the largest part size less than or equal to 50 MB or 50% of the client object size, whichever is smaller.
Client object size of 100 MB
Send client objects in 50 MB parts.
Client object size of 50 MB
Send client objects in 25 MB parts.
  • Network data loss is not a concern.
  • Multipart upload is done in parallel.
  • Multiple threads send the different parts of a front-end object in parallel.

Use smaller part sizes so that more threads can send data at the same time.

For example, when you use the S3 protocol transfer manager service, a thread pool of many threads can be used to transfer object parts in parallel.

For a thread pool of 10 threads and an object size of 100 MB, a part size of 10 MB can provide for optimal parallelism across threads. For parallel multipart upload, use a part size greater than or equal to the minimum of 5 MB. In this way, you can more effectively distribute the client object across all of the threads.