Using Filters to Include and Exclude Files

Filters refine the list of source files (or directories) to transfer by indicating which to skip or include based on name matching. When no filtering rules are specified by the client, Ascp transfers all source files in the transfer list; servers cannot filter client uploads or downloads.

Command Line Syntax

  • -E 'pattern'      Exclude files or directories with names or paths that match pattern.
  • -N 'pattern'      Include files or directories with names or paths that match pattern.

Where:

  • pattern is a file or directory name, or a set of names expressed with UNIX glob patterns.
  • Surround patterns that contain wildcards with single quotes to prevent filter patterns from being interpreted by the command shell. Patterns that do not contain wildcards can also be in single quotes.

Basic usage

  • Filtering rules are applied to the transfer list in the order they appear on the command line. If filtering rules are configured in aspera.conf, they are applied before the rules on the command line.
  • Filtering is a process of exclusion, and -N rules override -E rules that follow them. -N cannot add back files that are excluded by a preceding exclude rule.
  • An include rule must be followed by at least one exclude rule, otherwise all files are transferred because none are excluded. To exclude all files that do not match the include rule, use -N '/**/' -E '/**' at the end of your filter arguments.
  • Filtering operates only on the set of files and directories in the transfer list. An include rule (-N) cannot add files or directories that are not already part of the transfer list.
Example Transfer Result
-E 'rule' Transfer all files and directories except those with names that match rule.
-N 'rule' Transfer all files and directories because none are excluded.

To transfer only the files and directories with names that match rule use:

ascp -N 'rule' -N '/**/' -E '/**'
-N 'rule1' -E 'rule2' Transfer all files and directories with names that match rule1, as well as all other files and directories except those with names that match rule2.
-E 'rule1' -N 'rule2' Transfer all files and directories except those with names that match rule1. All files and directories not already excluded by rule1 are included because no additional exclude rule follows -N 'rule2'.

To transfer only the files and directories with names that do not match rule1 but do match rule2 use:

ascp -E 'rule1' -N 'rule2' -N '/**/' -E '/**'

Filtering Rule Application

Filters can be specified on the ascp command line and in aspera.conf. Ascp applies filtering rules that are set in aspera.conf before it applies rules on the command line.

Filtering order

Filtering rules are applied to the transfer list in the order they appear on the command line.

  1. Ascp compares the first file (or directory) in the transfer list to the pattern of the first rule.
  2. If the file matches the pattern, Ascp includes it (-N) or excludes it (-E) and the file is immune to any following rules.
    Note: When a directory is excluded, directories and files in it are also excluded and are not compared to any following rules. For example, with the command-line options -E '/images/' -N '/images/icons/', the directory /images/icons/ is not included or considered because /images/ was already excluded.
  3. If the file does not match, Ascp compares it with the next rule and repeats the process for each rule until a match is found or until all rules have been tried.
  4. If the file never matches any exclude rules, it is included in the transfer.
  5. The next file or directory in the transfer list is then compared to the filtering rules until all eligible files are evaluated.

Example

Consider the following command:

# ascp -N 'file2' -E 'file[0-9]' /images/icons/ user1@examplehost:/tmp

Where /images/icons/ is the source.

If /images/icons/ contains file1, file2, and fileA, the filtering rules are applied as follows:

  1. file1 is compared with the first rule (-N 'file2') and does not match so filtering continues.
  2. file1 is compared with the second rule (-E 'file[0-9]) and matches, so it is excluded from the transfer.
  3. file2 is compared with the first rule and matches, so it is included in the transfer and filtering stops for file2.
  4. fileA is compared with the first rule and does not match so filtering continues.
  5. fileA is compared with the second rule and does not match. Because no rules exclude it, fileA is included in the transfer.
    Note: If the filtering rules ended with -N '/**/' -E '/**', then fileA would be excluded because it was not "protected" by an include rule.

Rule Patterns

Rule patterns (globs) use standard globbing syntax that includes wildcards and special characters, as well as several Aspera extensions to the standard.

  • Character case: Case always matters, even if the file system does not enforce such a distinction. For example, on Windows FAT or NTFS file systems and macOS HPFS+, a file system search for "DEBUG" returns files "Debug" and "debug". In contrast, Ascp filter rules use exact comparison, such that "debug" does not match "Debug". To match both, use "[Dd]ebug".
  • Partial matches: With globs, unlike standard regular expressions, the entire filename or directory name must match the pattern. For example, the pattern abc*f matches abcdef but not abcdefg.

Standard Globbing: Wildcards and Special Characters

/ The only recognized path separator.
\ Quotes any character literally, including itself. \ is exclusively a quoting operator, not a path separator.
* Matches zero or more characters, except "/" or the . in "/.".
? Matches any single character, except "/" or the . in "/.".
[ … ] Matches exactly one of a set of characters, except "/" or the . in "/.".
[^… ] When ^ is the first character, matches exactly one character not in the set.
[!… ] When ! is the first character, matches exactly one character not in the set.
[x-x] Matches exactly one of a range of characters.
[:xxxxx:] For details about this type of wildcard, see any POSIX-standard guide to globbing.

Globbing Extensions: Wildcards and Special Characters

no / or * at end of pattern Matches files only.
/ at end of pattern Matches directories only. With -N, no files under matched directories or their subdirectories are included in the transfer. All subdirectories are still included, although their files will not be included. However, with -E, excluding a directory also excludes all files and subdirectories under it.
* or /** at end of pattern Matches both directories and files.
/** Like * but also matches "/" and the . in "/.".
/ at start of pattern Must match the entire string from the root of the transfer set. (Note: The leading / does not refer to the system root or the docroot.)

Standard Globbing Examples

Wildcard Example Matches Does Not Match
/ abc/def/xyz abc/def/xyz abc/def
\ abc\? abc? abc\? abc/D abcD
* abc*f abcdef abc.f abc/f abcefg
? abc?? abcde abc.z abcdef abc/d abc/.
[ … ] [abc]def adef cdef abcdef ade
[^… ] [^abc]def zdef .def 2def bdef /def /.def
[!… ] [!abc]def zdef .def 2def cdef /def /.def
[:xxxxx:] [[:lower:]]def cdef ydef Adef 2def .def

Globbing Extension Examples

Wildcard Example Matches Does Not Match
/** a/**/f a/f a/.z/f a/d/e/f a/d/f/ za/d/f
* at end of rule abc* abc/ abcfile  
/** at end of rule abc/** abc/.file abc/d/e/ abc/
/ at end of rule abc/*/ abc/dir abc/file
no / at end of rule abc abc (file) abc/
/ at start of rule /abc/def /abc/def xyz/abc/def

Testing Your Filter Rules

If you plan to use filtering rules, it's best to test them first. An easy way to test filtering rules, or to learn how they work, is to set up source and destination directories and use demo.asperasoft.com as the Aspera server:

  1. On your computer, create a set of directories and files (size can be small) that approximate a typical transfer file set. In the following example, the file set is in /tmp/src.
  2. Upload the file set to the Aspera demo server (demo.asperasoft.com) with the following command:
    # ascp /tmp/src aspera@demo.asperasoft.com:Upload/

    Where the user is "aspera" and the target is the Upload directory. At the prompt, enter the password "demoaspera".

  3. Create a destination directory on your computer, for example /tmp/dest.
  4. Download your files from the demo server to /tmp/dest to test your filtering rules. For example: 
    # ascp -N 'wxy/**' -E 'def' aspera@demo.asperasoft.com:Upload/src/ /tmp/dest
  5. Compare the destination directory with the source to determine if the filter behaved as expected.
    $ diff -r dest/ src/

    The diff output shows the missing files and directories (those that were not transferred).

Example Filter Rules

The example rules below are based on running a command such as the following to download a directory AAA from demo.asperasoft.com to /tmp/dest:

# ascp rules aspera@demo.asperasoft.com:Upload/AAA /tmp/dest

The examples below use the following file set:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile

Key for interpreting example results below:

< xxx/yyy = Excluded
xxx/yyy = Included
zzz/ = directory name
zzz = filename

  1. Transfer everything except files and directories starting with ".":
    -N '*' -E 'AAA/**'

    Results:

    AAA/abc/def
AAA/abc/wxy/def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/.def
  2. Exclude directories and files whose names start with wxy:
    -E 'wxy*'

    Results:

    AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile
  3. Include directories and files that start with "wxy" if they fall directly under AAA:
    -N 'wxy*' -E 'AAA/**'

    Results:

    AAA/wxy/
AAA/wxyfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxy/xyx/
< AAA/wxy/xyxfile
  4. Include directories and files at any level that start with wxy, but do not include dot-files, dot-directories, or any files under the wxy directories (unless they start with wxy). However, subdirectories under wxy will be included:
    -N '*/wxy*' -E 'AAA/**'

    Results:

    AAA/abc/wxy/tuv/
AAA/abc/xyz/def/wxy
AAA/wxyfile 
AAA/wxy/xyx/
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def     * 
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/wxy/xyxfile

    * Even though wxy is included, def is excluded because it's a file.

  5. Include wxy directories and files at any level, even those starting with ".":
    -N '*/wxy*' -N '*/wxy/**' -E 'AAA/**'

    Results:

    AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
  6. Exclude directories and files starting with wxy, but only those found at a specific location in the tree:
    -E '/AAA/abc/wxy*'

    Results:

    AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
  7. Include the wxy directory at a specific location, and include all its subdirectories and files, including those starting with ".":
    -N 'AAA/abc/wxy/**' -E 'AAA/**'

    Results:

    AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile