Transferring and deleting files with the Aspera Watch Service

When used along with ascp commands, the Aspera® Watch Service (asperawatchd) allows for fast detection and sending of new and deleted items. By comparing snapshots of the file directory it is watching, asperawatchd generates file lists for ascp transfers.

Prerequisites

To generate snapshots and file lists, configure and start asperawatchd. For more information, see Configuring the Aspera Watch Service.

Creating a subscription, snapshots, and snapshot differential

Procedure

Create a subscription and decide how to manage its expiration.
```
$ /Library/Aspera/bin/aswatchadmin subscribe daemon filepath [options]
```
By default, subscriptions expire in 24 hours. If your snapshot comparisons will be spaced more than 24 hours apart, either set the expiration time to a duration longer than the time between snapshots (add --expire_in=seconds to the command) or send a resubscribe command periodically to maintain the subscription.

For more information about creating subscriptions and resubscribing to them, see Managing Watch subscriptions.

In the following example, user aspera subscribes to /projectA/source and the subscription expires in 48 hours:
```
$ /Library/Aspera/bin/aswatchadmin subscribe aspera /projectA/source --expire_in=172800
[aswatchadmin subscribe] Successfully created subscription {"identifier":"bec581b3-3c34-47d7-a719-93f26f8272d1","path":"file:////projectA/source","scan_period":{"sec":9223372036854775807,"usec":999999},"expiration":"2018-03-15T07:39:21Z"}
 
```
Record the subscription ID (the value of "identifier" in the output) for use in creating the snapshot. You can also retrieve the subscription ID later.

Create a snapshot.

$ /Library/Aspera/bin/aswatchadmin create-snapshot daemon subscription_id

If you do not have the subscription ID, run the following command:

$ /Library/Aspera/bin/aswatchadmin query-subscriptions daemon

In the following example, user aspera creates a snapshot of the directory that is watched by subscription bec581b3-3c34-47d7-a719-93f26f8272d1:

$ /Library/Aspera/bin/aswatchadmin create-snapshot aspera bec581b3-3c34-47d7-a719-93f26f8272d1
[aswatchadmin create-snapshot] Successfully created snapshot 1.

After the wanted interval, create another snapshot to compare with the previous snapshot.
The snapshot ID is automatically incremented with each create-snapshot command.
For example, running the same command as the previous step outputs a new snapshot:
```
$ /Library/Aspera/bin/aswatchadmin create-snapshot aspera bec581b3-3c34-47d7-a719-93f26f8272d1
[aswatchadmin create-snapshot] Successfully created snapshot 2.
```
Generate the snapshot differential between the most recent snapshot and the snapshot before it.
To create a snapshot differential that outputs a list that can be used by ascp, run the following command:
```
$ /Library/Aspera/bin/aswatchadmin snapshot-differential daemon subscription_id snapshot_id --format=PATH 
```
Where the snapshot ID is the latest snapshot.
For example,
```
$ /Library/Aspera/bin/aswatchadmin snapshot-differential aspera bec581b3-3c34-47d7-a719-93f26f8272d1 2
/new_file.png
/new_file.pdf
```
Save the file list for use in the transfer session.

Send the new and modified files with ascp or ascp4.

Use the --source-prefix option to append the watch directory path to the file paths in the list:

$ ascp --file-list=filelist_pathname --source-prefix=prefix --mode=send --user=username --host=host target_directory

For example,

$ ascp --file-list=/Users/aspera/filelist.txt --source-prefix=/projectA/source --mode=send --user=aspera --host=10.0.0.1 /projectA/destination 
new_file.png       100%   10MB    9.7Mb/s     00:07    
new_file.pdf       100%   100MB   9.7Mb/s     00:35    
Completed: 112640K bytes transferred in 42 seconds
 (268190 bits/sec), in 2 files.

Results

Removing files from the target directory

The asdelete utility compares the source directory with the target directory and deletes extraneous files from the target directory. Run first with the -d option to do a dry run and view a list of files that would be deleted in an actual run. If the initiator of the asdelete command is a Windows™ OS, files that contain ASCII characters (such as < | ? ") are not deleted and an error is logged.

CAUTION: asdelete follows symbolic links, which can result in files that are being deleted and that are not within the target directory.

$ asdelete --host host --auth-name username --auth-pass password /source_directory /target_directory

$ /Library/Aspera/bin/asdelete --host host --auth-name username --auth-pass password /source_directory /target_directory

For example,

$ /Library/Aspera/bin/asdelete --host 10.0.0.1 --auth-name aspera --auth-pass !XF345lui@0 /projectA/source /projectA/destination

View the target directory to confirm deletion of the correct files.