Creating a pull Watch Folder with the API

These instructions describe how to create a pull Watch Folder by using the Watch Folder API.

You can also create and manage Watch Folders from the command line (Creating a pull Watch Folder with aswatchfolderadmin) or by using IBM Aspera Console.

When you create a Watch Folder, a Watch service subscription is automatically created to monitor the source directory. In the rare case that the subscription is somehow deleted or impaired, Watch Folders automatically creates a new subscription; however, the new subscription does not retain the file change history and all files in the source directory are retransferred.

Restrictions on all Watch Folders
  • Only local to remote (push) and remote to local (pull) configurations are supported. Remote to remote and local to local are not supported.
  • Growing files are only supported for local sources (push Watch Folders) and must be authenticated by a transfer user (password or SSH key file). The transfer user cannot be restricted to aspshell and the source cannot be in object storage. Growing files are only supported for local sources and nonobject storage-based destinations.
  • Source file archiving is not supported if the Watch Folder source is in object storage.
  • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version.
Restrictions on Pull Watch Folders
  • The remote server must be running HSTS or HSTE.
  • Pull Watch Folders must be authenticated with an access key ID and secret, a Node API username and password, SSH keys, SSH username and password, or IBM Aspera Shares credentials. (V4.4.3)
  • Pull Watch Folders that use Node API authentication cannot be authenticated with a Node API user whose associated transfer user is configured with a restriction (the Watch Folder status is reported as impaired). Edit the transfer user's configuration to use a docroot, restart asperanoded, and the Watch Folder recovers automatically.
  • Pull Watch Folders cannot use IBM Aspera on Cloud (including IBM Aspera on Cloud transfer service nodes) or IBM Aspera Transfer Cluster Manager nodes as the remote source.
  • Pull Watch Folders do not support growing files.
  1. Prepare your computer as described in Getting started with Watch Folders from the command line.
  2. Create a Node API user and associate it with a transfer user account. The user account must have administrative (root or sudo) privileges to interact with asperawatchfolderd.
    # /opt/aspera/bin/asnodeadmin -a -u node_username -p node_password -x admin_user --acl-set "admin,impersonation"

    For example,

    # /opt/aspera/bin/asnodeadmin -a -u watchfolder_user -p X245lskd3 -x root --acl-set "admin,impersonation"

    Adding, modifying, or deleting a node user triggers automatic reloading of the user database and the node's configuration and license files. For more information about the Node API, see your transfer server's administrator guide.

  3. Verify that you correctly added the Node API user.
    # /opt/aspera/bin/asnodeadmin -l
    List of Node API user(s):
                    user       system/transfer user                    acls
    ====================    =======================    ====================
           node_api_user                system_user    [admin,impersonation]

    For example, by using the information from the example in the previous step, the output is similar to the following:

    # /opt/aspera/bin/asnodeadmin -l
                    user       system/transfer user                    acls
    ====================    =======================    ====================
    watchfolder_user                          root    [admin,impersonation]
  4. Create a Watch Service on the remote server.
    This approach requires that you have node credentials for the remote server.
    1. Create a JSON configuration file for the remote Watch Service.
      {
          "type": "WATCHD",
          "run_as": {
              "user": "username",
              "pass": "password"
          },
          "enabled": true
      } 

      The username and password are for a transfer user with permissions to the source path. Save the file as wfd_create.json.

    2. To create the service, run the following command:
      # curl -ki -u node_username:node_password -X POST -d @wfd_create.json "https://server_ip_address:9092/rund/services"

      The output includes the service ID. Record the ID for the next substep.

    3. Confirm that the service is running.
      # curl -ki -u node_username:node_password -X GET "https://server_ip_address:9092/rund/services/service_id"
  5. Create the Watch Folder service on the local computer.
    1. Create a JSON configuration file for the service with the following text:
      {
          "type": "WATCHFOLDERD",
          "run_as": {
              "user": "username",
              "pass": "password"
          },
          "enabled": true
      } 

      The username and password are for a transfer user with permissions to the source path. Save the files, with the .json extension.

    2. Create the service.
      # curl -ki -u node_username:node_password -X POST -d @config_file "https://localhost:9092/rund/services"

      If service creation succeeds, the ID of the service is returned. Record the ID for use in the next step.

    3. Confirm that the service is running.
      # curl -ki -u node_username:node_password -X GET "https://localhost:9092/rund/services/service_id"
  6. Create a JSON configuration file for your Watch Folder.
    The Watch Folder JSON file describes the source, target, and authentication to the remote server, and can also specify transfer session settings, file handling and post-processing, filters, and growing file handling.

    A basic pull Watch Folder configuration has the following syntax:

    {
        "source": {
            "path": "source_directory",
            "location": {
                "type": "REMOTE",
                "host": "ip_address",
                "port": port,
                "authentication": {
                    "type": "authentication_mode",
                    "user": "username",
                    "pass": "password"
                 }
            }
        },
        "target": {
            "path": "target_directory"
        },
        "watchd": {
            "scan_period": "scan_period",
            "identifier": "daemon"
        }
    }

    For a full configuration reference, see Watch Folder JSON configuration file reference.

    Field Description Default
    source path The source directory on the remote server. For SSH and Node API user authentication, the path is relative to the associated transfer user's docroot, or the absolute path if the transfer user is configured with a restriction. For Shares authentication, the path is the share name and optionally, a path within the share. For access key authentication, the path is relative to the storage specified in the access key. N/A
    location type Set "type" to "REMOTE" for the remote server. "type": "REMOTE" is assumed when "host" is specified. "REMOTE"
    host The host IP address, DNS, hostname, or URL of the remote file system. Required. The host can be specified with an IPv4 or IPv6 address. The preferred format for IPv6 addresses is x:x:x:x:x:x:x:x, where each of the eight x is a hexadecimal number of up to 4 hex digits. Zone IDs (for example, %eth0) can be appended to the IPv6 address. N/A
    port The port to use for authentication to the remote file system. By default, if the authentication type is SSH, then the SSH port for the ascp process (the value for tcp_port in the "transport" section) is used. If the authentication type is NODE_BASIC, 9092 is used. For Shares, IBM Aspera Transfer Cluster Manager, or IBM Aspera on Cloud endpoints, enter 443. If authentication type is SSH, then default is the value for tcp_port in the "transport" section (default: 22). If authentication type is NODE_BASIC, then default is 9092.
    authentication type How Watch Folders authenticates to the remote server. Pull Watch Folders must use NODE_BASIC and authenticate with a Node API username and password, Shares credentials, or an access key ID and secret. NODE_BASIC
    user The username for authentication. Required. Depending on the type of authentication, it is the transfer user's username, Node API username, Shares username, or access key ID. N/A
    pass The password for authentication, depending on the type of authentication. N/A
    target path The target directory on the local computer, relative to the transfer user's docroot. N/A
    watchdidentifier The daemon associated with the Watch Service that is used to monitor the file system. Optional. Required only when you want to use a Watch Service that is run by a user who is not associated with the Node API user or access key. The system user that is associated with the Node API user or access key.
    scan_period The time between file system scans of the watches (from end of one to start of the next). These scans are independent of the snapshot minimum interval and snapshot minimum changes to ensure that changes are identified. To never scan (asperawatchd relies entirely on file notifications), set to "infinite". On file systems without file notifications, such as object storage, mounted storage (NFS), and AIX file system scans triggered by the scan period are used to detect file changes. In this case, set the scan period to frequently scan for changes. On operating systems that support file notifications (Linux, Windows, macOS), asperawatchd uses the file notifications as the primary means for detecting changes, and the scan period serves as a backup. In this case, the default value of 30 minutes is usually acceptable and no change is necessary. To never scan, and rely entirely on file notifications, set to "infinite".

    For pull Watch Folders, file systems scans that are triggered by scan_period are the sole means for detecting changes in the source directory.

    The lower scan periods detect changes faster but can result in greater resource consumption, particularly for object storage.

    Note: The value for scan period cannot be empty, otherwise the configuration is rejected.
    30 m

    Save the configuration file. The path to the configuration file is used in the next step.

  7. Start the Watch Folder.
    # curl -k --user node_api_user:node_api_password -H "X-aspera-WF-version:2017_10_23" -X POST -d @path/to/json_file https://host:node_api_port/v3/watchfolders

    By default, the API port is 9092.

    Note: The header "X-aspera-WF-version:2017_10_23" is required when submitting POST, PUT, and GET requests to /v3/watchfolders on servers. These enables Watch Folders to parse the JSON "source" and "target" objects in the format that was introduced.
    For example,
    # curl -k --user watchfolder_admin:XF324cd28 -H "X-aspera-WF-version:2017_10_23" -X POST -d @/watchfolder_conf.json https://198.51.100.22:9092/v3/watchfolders
    {
    "id": "b394d0ee-1cda-4f0d-b785-efdc6496c585"
    }
  8. Verify that the Watch Folder is running.
    # curl -k --user node_api_user:node_api_password -H "X-aspera-WF-version:2017_10_23" -X GET https://host:node_api_port/v3/watchfolders/watchfolder_id/state
    For example,
    # curl -sk --user watchfolder_admin:XF324cd28  -H "X-aspera-WF-version:2017_10_23" -X GET https://198.51.100.22:9092/v3/watchfolders/b394d0ee-1cda-4f0d-b785-efdc6496c585/state

    If the Watch Folder is running, it is reported with "state":"HEALTHY"

You can manage Watch Folders by using the API. For more information, see Managing Watch Folders with the API.