Managing storage usage

Edit online

There are several strategies you can use to manage storage. Some strategies are based on how you configure your Aspera on Cloud organization. Some are based on how to configure the storage you attach to your organization.

To understand storage in your subscription and billing, see Manage storage in your subscription.

Bring your own storage

Edit online

You can attach your own storage to your Aspera on Cloud organization.

If your existing storage is in a supported region in IBM Cloud, AWS S3, GCS, or Azure, you can access it through Aspera on Cloud.

Attach your storage using the procedures listed in this document: Attaching Cloud Storage to Your AoC Organization. Then, when you create a new workspace, select a location on your attached storage for the content in the workspace Files and Packages apps. These storage costs are then part of your normal cloud bill. In this scenario, you can also configure lifecycle policies from your cloud provider to remove files automatically. For more details and strategies, see the following section, "Configure Your Cloud Directory Structure".

Configure your cloud directory structure

Edit online

You can configure your directory structure to take maximum advantage of cloud provider lifecycle policies to manage your storage.

As a bit of background, the Aspera on Cloud default directory structure allows users to proceed as quickly as possible to efficient usage and collaboration. The default directory structure is as follows:
'''
.
   /
      <workspace_id>/
         home/
            <user_email_address + user_id>
               foo/
                  bar
      packages/
         <package_id>/
            foo/
               bar
'''
However, in cloud object storage, file storage is a large, flat directory with file name prefixes that Aspera and other software use to assume the intended directory structure. Cloud object storage for an Aspera workspace is as follows:
'''
.
   /<workspace_id>/
   /<workspace_id>/home/
   /<workspace_id>/home/<user_email_address + user_id>/
   /<workspace_id>/home/<user_email_address + user_id>/foo/
   /<workspace_id>/home/<user_email_address + user_id>/foo/bar
   /<workspace_id>/packages/
   /<workspace_id>/packages/<package_id>/
   /<workspace_id>/packages/<package_id>/foo/
   /<workspace_id>/packages/<package_id>/foo/bar
'''

In cloud object storage, directories are objects subject to deletion through lifecycle policies. It's vital to configure such policies so as not to delete folders that Aspera on Cloud requires for a workspace to function properly. For example, if you delete /<workspace_id>/packages/, package sending would be impossible. If you delete a user's home folder, /<workspace_id>/home/, that user would not have a working Files app.

A useful strategy is to centralize your files so that you can configure lifecycle policies with a prefix that will not disrupt the required folder structure.

Consider the following structure:
'''
.
   /
      /Shared/
         foo/
            bar
      <workspace_id>/
         home/
            <user_email_address + user_id>/
               Shared.asp-link/
'''

Such a structure centralizes the content your users need in the /Shared/ directory, outside the workspace folder.

To centralize your workspace content in the /Shared/ directory, you can share folders with workspace members using the Admin app (rather than the Files app). This allows users to collaborate in folders that are not in the workspace structure. When you share from the Admin app, workspace members access shared content by means of symlinks that display in their Files app home folder. In your cloud storage, these symlinks appear as very small files with the suffix .asp-link.

When you set cloud lifecycle policies, apply deletion only to files with the prefix /Shared/; this configuration prevents deletion of directories that disrupt Aspera on Cloud workspace operations.

Prevent users from uploading content to any other workspace area by configuring the Files app home folder to be read-only; for details and a procedure, see the following section, "Centralize Storage in the Files App".

Configure file deletion policies

Edit online

You can create policies to permanently delete files and folders when they reach a configured age. If the policy deletes all files in a given folder, the policy deletes the folder as well. Existing empty folders are not deleted. Shared folders and folders with public links are never deleted by a file deletion policy, though the content they contain may be deleted.

Attention:
  • For ATS nodes: If there are additional nodes in the same bucket, the same file deletion policy applies to those nodes as well.
  • For tethered nodes: The policy applies only to the specific node that you are creating the policy for. Beware that if there are other nodes that share the same storage and bucket path, this policy is going to affect those as well. This could lead to content being deleted from other nodes that the policy was not intended to.
Note: You can set file deletion policies on buckets in AWS, S3, CGCS and Microsoft Azure storage.
  1. Go to Nodes and storage > Nodes. Click the row of the intended node.
  2. Go to Profile > File deletion policy.
  3. Click the toggle to On.
  4. Set the deletion age in days. Valid values are 7 to 3650 days. The deletion job runs daily. A timer is set to start 24 hours after the last run. It does not have a set start time, as it runs depending on when it was last completed.
    Note: The time the job takes to run depends on how large the bucket is.

    For example, if you configure the deletion age to be 90 days, the policy will:

    • Permanently delete files in that bucket 90 days after their creation. This deletion age applies to existing files that are older than the deletion age, as well as to new files when they reach the configured deletion age.
    • Permanently delete folders that are emptied by the deletion job.
    • Not delete empty folders that were not affected by the policy.
    • Not delete shared folders or folders with public links, even if they are emptied by the deletion job.
  5. Click Save to activate the file deletion policy.

Centralize storage in the Files app

Edit online

By default, users can create folders and upload files in the root level of their Files app. Others cannot see that content until the user shares it with them.

This root level is also called the home folder. In the Aspera on Cloud interface, this area displays the label All files. The home folder includes folders created by the local user at the root level.

Screenshot showing the All Files area of the Files app. Folders shared by others have subtext displaying who shared the folder. Folders created the local user do not have that subtext.

Note: Admins can view and manage content in Files home folders if they have the node secret. See the section "Managing User Storage in the Workspace" below in this article. Admins can also access the cloud storage directly if they have the proper credentials.

But in any given workspace, you can prevent members from uploading content to the root level of their Files app. This restriction ensures that:

  • You can readily manage storage by applying cloud storage lifecycle policies to centralized storage in folders outside the workspace directory structure.
  • The storage your organization pays for contains only the contents you intend.
  • The transfer volume charged to your organization supports your corporate, team, or project mission, rather than being used for unrelated storage by individual members.

Whether or not you configure this restriction, you can share folders with members using the Admin app and grant them permission to view, upload to, or download from those folders. In this way, members can collaborate in centralized folders that you can readily monitor and manage, eliminating excess storage.

To restrict all users in a given workspace from uploading to and creating new folders in the root of their Files app, do the following:

  1. Go to Workspaces > workspace_name > Profile > Node.
  2. To restrict users of this workspace from creating new folders and from uploading content to the root level of their Files app, select the check box Workspace members cannot upload to workspace root level.
  3. Click Save.

To exempt specific users from the workspace restriction, do the following:

  1. Go to Workspaces > workspace_name >Member > memberName > Allow upload.
  2. Select the check box labeled Allow this user to upload to root level of workspace.
  3. Click Save.

Delete packages

Edit online

You can manually delete package content when it is no longer required.

Note: You can also configure policies that automatically delete packages when certain time-based or event-based thresholds occur; see Content Expiration Policies.

Deleting package content does not delete the package listing or package details -- only the assets in the package. You can still access details, including metadata, by clicking the package row and selecting More detail or Metadata.

There are two ways to accomplish this task:
  • Through Organization > Activity.
  • Through Nodes > Workspaces > Packages.

Through Organization > Activity

Note: The Activity page in the Organization section or in the Workspaces section of the Admin app is different than the AoC Activity application that can be made available to users with an Advanced tier subscription to Aspera on Cloud.
  1. Go to the Activity page:
    • Go to Organization > Activity.
    • Go to Workspaces > workspaceName > Activity.
  2. The table labeled Recent transfer activity displays recent package transfer event. Click View all in the lower left of the table to display an expanded, filterable list.
  3. To delete a package, do the following:
    1. Search or filter for the intended package, then click the package row to view details.
    2. Click the Actions (three dots) icon in the upper right of the package details panel, select Delete, then confirm Delete.

Through Nodes > Workspaces > Packages

  1. Go to Nodes > nodeName.
  2. In the Node access window, enter the node secret and click Log in.
  3. Click Workspaces, then click the row of the intended workspace.
  4. Click Packages.
  5. Browse, filter, or search for the intended package.
  6. To see package contents, click the package row to open the details panel.
  7. To delete package contents, click the Actions (three dots) menu at the top right of the details panel; select Delete, then confirm.

Manage folders on a node

Edit online

You can browse the node to create new folders and to inspect, rename, and delete content as required.

Note: You need the node secret or a valid custom access key/secret to complete this procedure.
Important: Use caution -- these actions can impact critical application files.

To monitor and manage general storage usage for a given node, do the following:

  1. Go to Nodes > nodeName > Browse.
  2. In the Node access window, enter the node secret and click Log in.
  3. Click Storage.
  4. Click a folder row to drill into it.
  5. To delete a folder on the node, browse to the intended folder, right-click the folder row, and select Delete.
  6. To rename a folder on the node, browse to the intended folder, right-click the folder row, and select Rename.
  7. To create a new folder on the node, click Create folder, name the new folder, and click OK.

Manage user storage in the workspace

Edit online

In a workspace with the Files app enabled, each standard user has their own root folder, containing the files and folders they upload and the folders shared with them by administrators or by other users.

In a workspace with the Packages app enabled, each standard user has their packages sent and received in this workspace.

As an administrator, you can do the following:

  • Files app
    • Delete the entire Files root folder of a deleted user.
    • Browse, rename, or delete individual items within a user's Files app.
    • Restrict users from uploading to the root level of their Files app.
  • Packages app
Note: You need the node secret or a valid custom access key and secret to complete this procedure. In this view, workspaces are identified by workspace ID; packages are identified by package ID.

To manage user workspace storage, do the following:

  1. Go to Nodes > nodeName > Browse.
  2. In the Node access window, enter the node secret and click Log in.
  3. Click Workspaces folder.
  4. Browse or filter for the intended workspace (by workspace ID), then click the row to open the workspace record.
  5. To manage content in a user's Files app:
    1. Click home; the list of users in the workspace displays, listed by registered email address.
    2. Browse or filter for the intended user; click the user row to display the content in that user's Files app.
      Note: Folders shared with this user appear with the link icon; folders uploaded by this user appear with the folder icon.
    3. If desired, sort the list by name or size by clicking the row heading.
    4. To rename a folder, right-click the folder row and select Rename.
    5. To unshare a shared folder, right-click the shared folder row and select Delete.
    6. To delete an unshared folder or content within it, go to the content to delete, right-click Delete.
  6. To manage content in a user's Packages app:
    1. Click packages ; the list of packages in the workspace displays, listed by package ID.
    2. To delete, select one or more packages, or click the package row to display its contents and select one or more items, then click Delete.