Creating a push Watch Folder with aswatchfolderadmin

These instructions describe how to create a push Watch Folder by using the aswatchfolderadmin utility. aswatchfolderadmin requires a JSON configuration file with a specific syntax. Push Watch Folders can still be created from JSON configuration files by using the Watch Folder API.

About this task

To create and manage Watch Folders by using the Watch Folder API, the GUI, or IBM® Aspera® Console, see Creating a push Watch Folder with the API, Watch Folders in the GUI, and IBM Aspera Console.

When you create a Watch Folder, a Watch service subscription is automatically created to monitor the source directory. In the rare case that the subscription is somehow deleted or impaired, Watch Folders automatically creates a new subscription; however, the new subscription does not retain the file change history and all files in the source directory are retransferred.

Restrictions on all Watch Folders
  • Only local to remote (push) and remote to local (pull) configurations are supported. Remote to remote and local to local are not supported.
  • Growing files are only supported for local sources (push Watch Folders) and must be authenticated by a transfer user (password or SSH key file). The transfer user cannot be restricted to aspshell and the source cannot be in object storage. Growing files are only supported for local sources and nonobject storage-based destinations.
  • Source file archiving is not supported if the Watch Folder source is in object storage.
  • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version.

To create a push Watch Folder:

Procedure

  1. Prepare your computer as described in Getting started with Watch Folders from the command line.
  2. Create a Watch Service and Watch Folder service for your user on the local computer. 
    $ /Library/Aspera/sbin/asperawatchd --user username
/Library/Aspera/sbin/asperawatchfolderd --user username
  3. Verify that the services are running for the user. 
    $ /Library/Aspera/bin/asrun send -l

    The output is similar to:

    [asrun send] code=0
{
    "services": [
         {
              "id":"52ca847a-6981-47e1-9f9b-b661cf298af1",
              "configuration": {
                     "enabled":true,
                     "run_as": {
                          "pass": "*****",
                          "user":"root"
                      },
                      "type":"WATCHD"
               },
               "state":"RUNNING",
               "state_changed_at":"2016-10-20T19:14:34Z"
          },
         {
               "id":"d109d1bd-7db7-409f-bb16-ca6ff9abb5f4",
               "configuration": {
                     "enabled":true,
                     "run_as":{
                          "pass": "*****",
                          "user":"root"
                      },
                      "type":"WATCHFOLDERD"
               },
               "state":"RUNNING",
               "state_changed_at":"2016-10-20T00:11:19Z"
           }
     ]
}
    Use the aswatchadmin and aswatchfolderadmin utilities to retrieve a list of running daemons. Daemons usually have the same name as the user for which they are running. For example, if you used the root user to run your services, you need see the root daemon that is listed when you run the following commands: 
    $ /Library/Aspera/bin/aswatchadmin query-daemons
[aswatchadmin query-daemons] Found a single daemon:
    root

 $ /Library/Aspera/bin/aswatchfolderadmin query-daemons
[aswatchfolderadmin query-daemons] Found a single daemon:
    root
    Note: Daemons for services that are started in the GUI are named with the ID of the service, rather than the user.
  4. Create a JSON configuration file for your Watch Folder.
    The Watch Folder JSON file describes the source, target, and authentication to the remote server, and can also specify transfer session settings, file handling and post-processing, filters, and growing file handling.

    A basic push Watch Folder configuration file has the following syntax:

    {
    "source": {
        "path": "source_directory"
    },
    "target": {
        "path": "target_directory",
        "location": {
            "type": "REMOTE",
            "host": "hostname",
            "port": port,
            "authentication": {
                "type": "authentication_mode",
                "user": "username",
                "pass": "password"
                "keypath": "key_file" 
              }
        }
    },
    "watchd": {
        "scan_period": "scan_period"
    }
}

    For a full configuration reference, see Watch Folder JSON configuration file reference.

    Field Description Default
    source path The local source directory. If the transfer user who is associated with the Node API user is configured with a docroot, then the path is relative to that docroot. If the transfer user is configured with a restriction, then the path is the absolute or UNC path. N/A
    target path The remote target directory. For SSH and Node API user authentication, the path is relative to the user's docroot, or the absolute path if the transfer user is configured with a restriction. For Shares authentication, the path is the share name and optionally, a path within the share. For access key authentication, the path is relative to the storage specified in the access key. N/A
    location type Set "type" to "REMOTE" for the remote server. "type": "REMOTE" is assumed when "host" is specified. "REMOTE"
    host The host IP address, DNS, hostname, or URL of the remote file system. Required. The host can be specified with an IPv4 or IPv6 address. The preferred format for IPv6 addresses is x:x:x:x:x:x:x:x, where each of the eight x is a hexadecimal number of up to 4 hex digits. Zone IDs (for example, %eth0) can be appended to the IPv6 address. N/A
    port The port to use for authentication to the remote file system. By default, if the authentication type is SSH, then the SSH port for the ascp process (the value for tcp_port in the "transport" section) is used. If the authentication type is NODE_BASIC, 9092 is used. For Shares, IBM Aspera Transfer Cluster Manager, or IBM Aspera on Cloud endpoints, enter 443. If authentication type is SSH, then default is the value for tcp_port in the "transport" section (default: 22). If authentication type is NODE_BASIC, then default is 9092.
    authentication type How Watch Folders authenticates to the remote server. Valid values are SSH or NODE_BASIC.

    For SSH, authenticate with a transfer user's username and password, or specify the username and the path to their SSH private key file.

    For NODE_BASIC, authenticate with a Node API username and password, Shares credentials, or an access key ID and secret.

    Sample JSON syntax for each authentication type is provided following this table.

    		 NODE_BASIC
    user The username for authentication. Required. Depending on the type of authentication, it is the transfer user's username, Node API username, Shares username, or access key ID. N/A
    pass The password for authentication. Depending on the type of authentication, it is the transfer user's password, the Node API user's password, the Shares user's password, or the access key secret.

    Required for SSH authentication if "keypath" is not specified.

    		 N/A
    keypath For SSH authentication with an SSH key, the path to the transfer user's SSH private key file.

    Required for SSH authentication if "pass" is not specified.

    		 N/A
    watchdidentifier The daemon associated with the Watch Service that is used to monitor the file system. Optional. Required only when you want to use a Watch Service that is run by a user who is not associated with the Node API user or access key. N/A
    scan_period The time between file system scans of the watches (from end of one to start of the next). These scans are independent of the snapshot minimum interval and snapshot minimum changes to ensure that changes are identified. To never scan (asperawatchd relies entirely on file notifications), set to "infinite". On file systems without file notifications, such as object storage, mounted storage (NFS), and AIX® file system scans triggered by the scan period are used to detect file changes. In this case, set the scan period to frequently scan for changes. On operating systems that support file notifications (Linux®, Windows™, macOS), asperawatchd uses the file notifications as the primary means for detecting changes, and the scan period serves as a backup. In this case, the default value of 30 minutes is usually acceptable and no change is necessary. To never scan, and rely entirely on file notifications, set to "infinite".

    For pull Watch Folders, file systems scans that are triggered by scan_period are the sole means for detecting changes in the source directory.

    The lower scan periods detect changes faster but can result in greater resource consumption, particularly for object storage.

    Note: The value for scan period cannot be empty, otherwise the configuration is rejected.
    		 30 m

    Save the configuration file. The path to the configuration file is used in the next step.

  5. Create the Watch Folder. 
    $ /Library/Aspera/bin/aswatchfolderadmin create-folder daemon -f json_file
    Where daemon is the user that is running the Watch Folder services and json_file is the path to the Watch Folder configuration file. If you do not know the daemon, retrieve a list of running daemons by running the following command: 
    $ /Library/Aspera/bin/aswatchfolderadmin query-daemons
[aswatchfolderadmin query-daemons] Found a single daemon:
    root
    Daemons usually have the same name as the user for which they are running. For example, if you used the root user to run your services, you need to see the root daemon listed.
    Note: Daemons for services that are started in the GUI are named with the ID of the service, rather than the user.

    For example, by using the root daemon and a valid JSON file, watchfolderconf.json, the output of the aswatchfolderadmin command needs to look like the following:

    $ /Library/Aspera/bin/aswatchfolderadmin create-folder root -f watchfolder_conf.json
                            [aswatchfolderadmin create-folder] Successfully created instance b394d0ee-1cda-4f0d-b785-efdc6496c585.

    If aswatchfolderadmin returns err=28672, confirm that the user's docroot allows access to the source directory. If you need to make changes to your docroot, see Updating the docroot or restriction of a running Watch Folder service.

    If aswatchfolderadmin returns err=2, a Watch Service is not running for the user. See the previous section for instructions on starting a Watch Service.

  6. Verify that the Watch Folder is running.
    To retrieve a list of running Watch Folders, run the following command: 
    $ /Library/Aspera/bin/aswatchfolderadmin query-folders daemon_name

    For example,

    $ /Library/Aspera/bin/aswatchfolderadmin query-folders root
                            [aswatchfolderadmin query-folders] Found a single watchfolder:
                            b394d0ee-1cda-4f0d-b785-efdc6496c585
  7. Test your Watch Folder.
    If the source directory is empty, add files to it. If the configuration is correct, Watch Folders detects the new files, starts a transfer, and the new files appear in the target directory.

    If the source directory is not empty, open the target directory to view files that are automatically transferred as Watch Folders starts.

Results

Once Watch Folders are created, manage them by using the aswatchfolderadmin utility. For more information, see Managing Watch Folders with aswatchfolderadmin.