Uploading objects in multiple parts

Edit online

When working with larger objects, multipart upload operations are recommended to write objects into IBM COS. An upload of a single object can be performed as a set of parts and these parts can be uploaded independently in any order and in parallel. Upon upload completion, IBM COS then presents all parts as a single object. This provides many benefits: network interruptions do not cause large uploads to fail, uploads can be paused and restarted over time, and objects can be uploaded as they are being created.

Multipart uploads are only available for objects larger than 5MB. For objects smaller than 50GB, 500 parts sized 20MB to 100MB is recommended for optimum performance. For larger objects, part size can be increased without significant performance impact. Multipart uploads are limited to no more than 10,000 parts of 5GB each and a maximum object size of 10TB.

Due to the additional complexity involved, it is recommended that developers make use of S3 API libraries that provide multipart upload support.

Incomplete multipart uploads do persist until the object is deleted or the multipart upload is aborted with AbortIncompleteMultipartUpload. If an incomplete multipart upload is not aborted, the partial upload continues to use resources. Interfaces should be designed with this point in mind, and clean up incomplete multipart uploads.

Standard conditional request headers (https://tools.ietf.org/html/rfc7232) are now supported for multipart uploads. They are only specified with the initiate request but are enforced during both initiate and complete processing. Any conditional request headers specified with PUT Part, Abort, and Complete requests will be ignored. If a Complete request fails due to a precondition error an Abort request must be sent to cleanup.

There are three phases to uploading an object in multiple parts:

The upload is initiated and an UploadId is created.
Individual parts are uploaded specifying their sequential part numbers and the UploadId for the object.
When all parts are finished uploading, the upload is completed by sending a request with the UploadId and an XML block that lists each part number and it’s respective Etag value.

Initiate a multipart upload

A POST issued to an object with the query parameter upload creates a new UploadId value, which is then be referenced by each part of the object being uploaded.

Syntax


POST https://{endpoint}/{bucket-name}/{object-name}?uploads= # path style
POST https://{bucket-name}.{endpoint}/{object-name}?uploads= # virtual host style

Specific headers for SSE-C

Common SSE-C headers are available for buckets using Server Side Encryption with Customer-Provided Keys (SSE-C) enabled.

Other headers

Upload an object with tagging by adding the x-amz-tagging header to an object upload request. For more information, see: Other headers.

Note: Release 3.19.5 and later: Prevents the overwrite of an existing object with a forbid overwrite precondition. For more information, see: Forbid Overwrite Header .

Object Lock headers

Object Lock Headers can be used during the Initiate request of a Multipart Upload.

Note: Non-bucket owners uploading objects to an Object Lock enabled bucket will have default retention of the bucket applied , if configured on the bucket. Otherwise no retention will be applied to the object.

Table 1. Object Lock headers
Name	Description	Required
x-amz-object-lock-mode	The object lock mode to be applied to the uploaded object. Type String Valid Values COMPLIANCE, GOVERNANCE Default None Note: Objects uploaded to an Object Lock enabled bucket with a default configuration rule OR uploaded with Object Lock specific headers require a Content-MD5 header (or use of V4 signature with content signing) or with a x-am-checksum- header included as a confirmation of request payload integrity.	Required if RetainUntilDate header is used; otherwise, no.
x-amz-object-lock-retain-until-date	The date and time when the uploaded object’s Object Lock is to expire. Type String Valid Values ISO8601 Date and Time Default None Note: Objects uploaded to an Object Lock enabled bucket with a default configuration rule OR uploaded with Object Lock specific headers require a Content-MD5 header (or use of V4 signature with content signing) or with a x-am-checksum- header included as a confirmation of request payload integrity.	Required if Mode header is used; otherwise, no.
x-amz-object-lock-legal-hold	The Legal Hold state to be applied to the object. Type String Valid Values ON \| OFF Default None Note: Objects uploaded to an Object Lock enabled bucket with a default configuration rule OR uploaded with Object Lock specific headers require a Content-MD5 header (or use of V4 signature with content signing) or with a x-am-checksum- header included as a confirmation of request payload integrity.	No

Sample request


POST /some-bucket/multipart-object-123?uploads= HTTP/1.1
Authorization: {authorization-string}
x-amz-date: 20170303T203411Z
x-amz-tagging: tag1=value1&tag2=value2
Host: 67.228.254.193
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>some-bucket</Bucket>
<Key>multipart-object-123</Key>
<UploadId>0000015a-95e1-4326-654e-a1b57887784f</UploadId>
</InitiateMultipartUploadResult>

Sample Object Lock request


POST /some-bucket/multipart-object-123?uploads= HTTP/1.1
Authorization: {authorization-string}
x-amz-object-lock-mode:COMPLIANCE 
x-amz-object-lock-retain-until-date : <ISO8601 date and time> 
x-amz-object-lock-legal-hold: ON 
Content-MD5: <MD5 of Contents>
Host: 67.228.254.193

Sample response


HTTP/1.1 200 OK
Date: Fri, 03 Mar 2017 20:34:12 GMT
X-Clv-Request-Id: 258fdd5a-f9be-40f0-990f-5f4225e0c8e5
Accept-Ranges: bytes
Server: Cleversafe/3.9.1.114
X-Clv-S3-Version: 2.5
Content-Type: application/xml
Content-Length: 276