These instructions describe how to create a pull Watch Folder by using the
aswatchfolderadmin utility. aswatchfolderadmin requires a JSON
configuration file with the syntax. Pull Watch Folders can still be created from JSON configuration
files that follow the 3.7 version syntax by using the Watch Folder API.
To create and manage Watch Folders by using the Watch Folder API or IBM Aspera Console, see Creating a push Watch Folder with the API and IBM Aspera
Console.
When you create a Watch Folder, a Watch service subscription is
automatically created to monitor the source directory. In the rare case that the subscription is
somehow deleted or impaired, Watch Folders automatically creates a new subscription; however, the
new subscription does not retain the file change history and all files in the source directory are
retransferred.
Restrictions on all Watch Folders
- Only local to remote (push) and remote to local (pull) configurations are supported. Remote to
remote and local to local are not supported.
- Growing files are only supported for local sources (push Watch Folders)
and must be authenticated by a transfer user (password or SSH key file). The transfer user cannot be
restricted to aspshell and the source cannot be in object storage. Growing files
are only supported for local sources and nonobject storage-based destinations.
- Source file archiving is not supported if the Watch Folder source is in object storage.
- IBM Aspera Shares endpoints must have version Shares version
1.9.11 with the Watch Folder patch or a later version.
Restrictions on Pull Watch Folders
- The remote server must be running HSTS or HSTE.
- Pull Watch Folders must be authenticated with an access key ID and secret, a Node API username
and password, SSH keys, SSH username and password, or IBM Aspera Shares credentials.
(V4.4.3)
- Pull Watch Folders that use Node API authentication cannot be authenticated with a Node API user
whose associated transfer user is configured with a restriction (the Watch Folder status is reported
as impaired). Edit the transfer user's configuration to use a docroot, restart asperanoded,
and the Watch Folder recovers automatically.
- Pull Watch Folders cannot use IBM Aspera on Cloud (including IBM Aspera on Cloud transfer service
nodes) or IBM Aspera Transfer Cluster Manager nodes as the remote source.
- Pull Watch Folders do not support growing files.
To create a pull Watch Folder:
-
Create a Watch Service on the remote server.
If you have SSH access to the server, create the service from the server's command
line.
-
Create the service.
# /opt/aspera/sbin/asperawatchd --user username
The
username is for a system user with permissions to the source path.
-
Confirm that the service was created.
# /opt/aspera/bin/aswatchadmin query-daemons
If the service exists, the following output is returned (in this example, the user is "root"):
# /opt/aspera/bin/aswatchadmin query-daemons
[aswatchadmin query-daemons] Found a single daemon:
root
If other services are running on the server, other daemons are also returned.
If you do not have SSH access to the server, use the Node API from your local computer to
create the service. This approach requires that you have node credentials for the server. For
instructions, see
Creating a pull Watch Folder with the API.
-
Create a Watch Folder service for your user on the local computer.
# /opt/aspera/sbin/asperawatchfolderd --user username
-
Verify that the service is running for the user.
# /opt/aspera/bin/asrun send -l
The output is similar to the following (in this example, the user is "root"):
[asrun send] code=0
{
"services": [
{
"id":"d109d1bd-7db7-409f-bb16-ca6ff9abb5f4",
"configuration": {
"enabled":true,
"run_as":{
"pass": "*****",
"user":"root"
},
"type":"WATCHFOLDERD"
},
"state":"RUNNING",
"state_changed_at":"2016-10-20T00:11:19Z"
}
]
}
Use the
aswatchadmin and
aswatchfolderadmin utilities to retrieve a list of running daemons. Daemons have the same name as the user for which they are running. For
example, if you used the
root user to run your services, you need see the
root daemon that is listed when you run the following
commands:
# /opt/aspera/bin/aswatchadmin query-daemons
[aswatchadmin query-daemons] Found a single daemon:
root
# /opt/aspera/bin/aswatchfolderadmin query-daemons
[aswatchfolderadmin query-daemons] Found a single daemon:
root
-
Create a JSON configuration file for your Watch Folder.
The Watch Folder JSON file describes the source, target, and authentication to
the remote server, and can also specify transfer session settings, file handling and
post-processing, filters, and growing file handling.
A basic pull Watch Folder configuration has
the following syntax:
{
"source": {
"path": "source_directory",
"location": {
"type": "REMOTE",
"host": "ip_address",
"port": port,
"authentication": {
"type": "authentication_mode",
"user": "username",
"pass": "password"
}
}
},
"target": {
"path": "target_directory"
},
"watchd": {
"scan_period": "scan_period",
"identifier": "daemon"
}
}
For a full configuration reference, see Watch Folder JSON configuration file reference.
Field |
Description |
Default |
source path |
The source directory on the remote server. For SSH and Node API user authentication, the
path is relative to the associated transfer user's docroot, or the absolute path if the transfer
user is configured with a restriction. For Shares authentication, the path is
the share name and optionally, a path within the share. For access key authentication, the path is
relative to the storage specified in the access key. |
N/A |
location type |
Set "type" to "REMOTE" for the remote server.
"type": "REMOTE" is assumed when "host" is specified. |
"REMOTE" |
host |
The host IP address, DNS, hostname, or URL of the remote file system. Required. The
host can be specified with an IPv4 or IPv6 address. The preferred format for IPv6 addresses is
x:x:x:x:x:x:x:x , where each of the eight x is a hexadecimal number of up to 4 hex
digits. Zone IDs (for example, %eth0) can be appended to the IPv6 address. |
N/A |
port |
The port to use for authentication to the remote file system. By default, if the
authentication type is SSH , then the SSH port for the ascp
process (the value for tcp_port in the "transport" section) is
used. If the authentication type is NODE_BASIC , 9092 is used. For Shares, IBM Aspera Transfer Cluster Manager, or IBM Aspera on Cloud endpoints, enter
443. |
If authentication type is SSH , then default is the value for
tcp_port in the "transport" section (default: 22). If
authentication type is NODE_BASIC , then default is 9092. |
authentication type |
How Watch Folders authenticates to the remote server. Pull Watch Folders must use
NODE_BASIC and authenticate with a Node API username and password, Shares credentials, or an access key ID and secret. |
NODE_BASIC |
user |
The username for authentication. Required. Depending on the type of
authentication, it is the transfer user's username, Node API username, Shares
username, or access key ID.
|
N/A |
pass |
The password for authentication, depending on the type of authentication. |
N/A |
target path |
The target directory on the local computer, relative to the transfer user's
docroot. |
N/A |
watchdidentifier
|
The daemon associated with the Watch Service that is used to monitor
the file system. Optional. Required only when you want to use a Watch Service that is run by a user
who is not associated with the Node API user or access key. |
The system user that is associated with the Node API user or
access key. |
scan_period |
The time between file system scans of the watches (from end of one
to start of the next). These scans are independent of the snapshot minimum interval and snapshot
minimum changes to ensure that changes are identified. To never scan (asperawatchd relies
entirely on file notifications), set to "infinite" .
On file systems without file notifications, such as object storage,
mounted storage (NFS), and AIX file system scans triggered by the scan period are used to detect
file changes. In this case, set the scan period to frequently scan for changes. On operating systems
that support file notifications (Linux, Windows, macOS), asperawatchd uses the file
notifications as the primary means for detecting changes, and the scan period serves as a backup. In
this case, the default value of 30 minutes is usually acceptable and no change is necessary. To
never scan, and rely entirely on file notifications, set to "infinite" .For pull Watch Folders, file systems scans that are triggered by scan_period are the
sole means for detecting changes in the source directory.
The lower scan periods detect
changes faster but can result in greater resource consumption, particularly for object
storage.
Note: The value for scan period cannot be empty, otherwise the configuration is
rejected.
|
30 m |
Save the configuration file. The path to the configuration file is used in the next
step.
-
Create the Watch Folder.
# /opt/aspera/bin/aswatchfolderadmin create-folder daemon -f json_file
Where
daemon is the user that is running the Watch Folder services and
json_file is the path to the Watch Folder configuration file. If you do not know
the daemon, retrieve a list of running daemons by running the following
command:
# /opt/aspera/bin/aswatchfolderadmin query-daemons
[aswatchfolderadmin query-daemons] Found a single daemon:
root
Daemons have
the same name as the user for which they are running. For example, if you used the
root user to run your services, you need to see the
root
daemon listed.
For example, by using the root daemon and a valid JSON file,
watchfolderconf.json, the output of the aswatchfolderadmin
command needs to look like the following:
# /opt/aspera/bin/aswatchfolderadmin create-folder root -f watchfolder_conf.json
[aswatchfolderadmin create-folder] Successfully created instance b394d0ee-1cda-4f0d-b785-efdc6496c585.
If aswatchfolderadmin returns err=28672
, confirm that the
user's docroot allows access to the source directory. If you need to make changes to your docroot,
see Updating the docroot or restriction of a running Watch Folder service.
If aswatchfolderadmin returns err=2
, a Watch Service is not
running for the user. See the previous section for instructions on starting a Watch Service.
-
Verify that the Watch Folder is running.
To retrieve a list of running Watch Folders, run the following
command:
# /opt/aspera/bin/aswatchfolderadmin query-folders daemon_name
For
example,
# /opt/aspera/bin/aswatchfolderadmin query-folders root
[aswatchfolderadmin query-folders] Found a single watchfolder:
b394d0ee-1cda-4f0d-b785-efdc6496c585
-
Test your Watch Folder.
If the source directory contains files, the Watch Folder collects them into a drop
after the Watch service scan interval passes and transfers them to the target.
Note: No files are
transferred until the first scan interval passes. If the Watch service scan interval is set to the
default, files transfer after 30 minutes.