Using Amazon S3 versioning with Aspera

The Amazon S3 bucket version feature gives you the option multiple versions of an object in the same bucket. By default, Aspera® uploads overwrite any object that has the same name as a source file.

Prerequisites

  • Versioning must be enabled on the Amazon S3 bucket. For instructions, see: Using versioning in S3 buckets.
  • Before upgrading from a previous version, make a backup and then delete /opt/aspera/etc/trapd/s3.properties. After upgrading, copy your customizations from the backup into the new s3.properties file that was created during the installation process.

Enabling S3 versioning with Aspera transfers

The configuration file /opt/aspera/etc/trapd/s3.properties contains the following properties that let Trapd detect if S3 versioning is enabled and changes the default behavior of Aspera uploads:

# The period after which a bucket is checked again for cross-region transfers or
# bucket versioning.
# Default is 900000 (15 minutes)
#aspera.session.check-bucket.frequency=900000

# Whether to retrieve the bucket versioning status. To retrieving the bucket
# versioning status one must be the owner of the bucket and have the permission
# "s3:GetBucketVersioning"
# Default is false
#aspera.session.retrieve-bucket-versioning=false

# The assumed bucket versioning status when aspera.session.retrieve-bucket-versioning is false.
# Default is false (assume bucket is not versioned).
#aspera.session.assumed-bucket-versioning=false

To enable S3 versioning:

  • If the user is the owner of the bucket, uncomment aspera.session.retrieve-bucket-versioning and set it to true.
  • If the user is not the owner of the bucket or does not have permission to check the bucket status, set or leave aspera.session.retrieve-bucket-versioning to false. Uncomment aspera.session.assumed-bucket-versioning and set it to true.

Aspera versioning behavior

File handling in a versioning-enabled Amazon S3 bucket depends on if the S3 versioning feature in Aspera is enabled.

With Aspera S3 versioning enabled:

  • When a file is first uploaded to the bucket, the object is created with a version ID.
  • When a source file is skipped during an upload because it exists in the bucket, the existing object and its ID do not change.
  • When an object is overwritten in the bucket, the new version of the object is created and assigned a version ID while the original object is left unmodified with the original ID.
    Note: The HSTS GUI does not display all the versions of an object. To view all the versions, use the Amazon S3 console or AWS SDKs.
  • When an object is deleted, the file initially disappears from the HSTS GUI; however, in the Amazon S3 console and AWS SDKs output, the object is replaced with a delete marker. This delete marker becomes the current object version. Previous versions are retained and upon refresh, the original object reappears in the HSTS GUI.

With Aspera S3 versioning disabled:

  • When a file is first uploaded to the bucket, the object is created with a version ID.
  • When an object is overwritten, the object is replaced by a delete marker and then a new version ID, indicating that the object was deleted and rewrote.