Using filters to include and exclude files

For example, on Windows™ FAT or NTFS (or directories) to transfer by indicating which to skip or include based on name matching. When no filtering rules are specified by the client, ascp transfers all source files in the transfer list.

Command line syntax

  • -E 'pattern'      Exclude files or directories with names or paths that match pattern.
  • -N 'pattern'      Include files or directories with names or paths that match pattern.

Where:

  • pattern is a file or directory name, or a set of names expressed with UNIX™ glob patterns.
  • Surround patterns that contain wildcards with single quotation marks (' ') to prevent filter patterns from being interpreted by the command shell. Patterns that do not contain wildcards can also be in single quotation marks (' ').

Basic usage

  • Filtering rules are applied to the transfer list in the order that they appear on the command line. If filtering rules are configured in aspera.conf, they are applied before the rules on the command line.
  • Filtering is a process of exclusion, and -N rules override -E rules that follow them. -N cannot add back files that are excluded by a preceding exclude rule.
  • An include rule must be followed by at least one exclude rule, otherwise all files are transferred because none are excluded. To exclude all files that do not match the include rule, use -N '/**/' -E '/**' at the end of your filter arguments.
  • Filtering operates only on the set of files and directories in the transfer list. An include rule (-N) cannot add files or directories that are not already part of the transfer list.
Example Transfer Result
-E 'rule' Transfer all files and directories except those with names that match rule.
-N 'rule' Transfer all files and directories because none are excluded.

To transfer only the files and directories with names that match rule use:

ascp -N 'rule' -N '/**/' -E '/**'
-N 'rule1' -E 'rule2' Transfer all files and directories with names that match rule1, and all other files and directories except those with names that match rule2.
-E 'rule1' -N 'rule2' Transfer all files and directories except those with names that match rule1. All files and directories that are not already excluded by rule1 are included because no additional exclude rule follows -N 'rule2'.

To transfer only the files and directories with names that do not match rule1 but do match rule2 use:

ascp -E 'rule1' -N 'rule2' -N '/**/' -E '/**'

Filtering rule application

Filters can be specified on the ascp command line and in aspera.conf. Ascp applies filtering rules that are set in aspera.conf before it applies rules on the command line.

Filtering order

Filtering rules are applied to the transfer list in the order that they appear on the command line.

  1. Ascp compares the first file (or directory) in the transfer list to the pattern of the first rule.
  2. If the file matches the pattern, Ascp includes it (-N) or excludes it (-E) and the file is immune to any following rules.
    Note: When a directory is excluded, directories and files in it are also excluded and are not compared to any following rules. For example, with the command line options -E '/images/' -N '/images/icons/', the directory /images/icons/ is not included or considered because /images/ was already excluded.
  3. If the file does not match, Ascp compares it with the next rule and repeats the process for each rule until a match is found or until all rules are tried.
  4. If the file never matches any exclude rules, it is included in the transfer.
  5. The next file or directory in the transfer list is then compared to the filtering rules until all eligible files are evaluated.

Example:

Consider the following command:

# ascp -N 'file2' -E 'file[0-9]' /images/icons/ user1@examplehost:/tmp

Where /images/icons/ is the source.

If /images/icons/ contains file1, file2, and fileA, the filtering rules are applied as follows:

  1. file1 is compared with the first rule (-N 'file2') and does not match so filtering continues.
  2. file1 is compared with the second rule (-E 'file[0-9]') and matches, so it is excluded from the transfer.
  3. file2 is compared with the first rule and matches, so it is included in the transfer and filtering stops for file2.
  4. fileA is compared with the first rule and does not match so filtering continues.
  5. fileA is compared with the second rule and does not match. Because no rules exclude it, fileA is included in the transfer.
    Note: If the filtering rules ended with -N '/**/' -E '/**', then fileA would be excluded because it was not protected by an include rule.

Rule patterns

Rule patterns (globs) use standard globbing syntax that includes wildcards and special characters, and several Aspera® extensions to the standard.

  • Character case: Case always matters, even if the file system does not enforce such a distinction. For example, on Windows FAT or NTFS file systems and macOS HPFS+ a file system search for DEBUG returns files Debug and debug. In contrast, ascp filter rules use exact comparison, such that debug does not match Debug. To match both, use [Dd]ebug.
  • Partial matches: With globs, unlike standard regular expressions, the entire file name or directory name must match the pattern. For example, the pattern abc*f matches abcdef but not abcdefg.

Standard globbing: Wildcards and special characters

/ The only recognized path separator.
\ Quotes any character literally, including itself. \ is exclusively a quoting operator, not a path separator.
* Matches zero or more characters, except / or the . in /..
? Matches any single character, except / or the . in /..
[ … ] Matches exactly one of a set of characters, except / or the . in /..
[^… ] When ^ is the first character, matches exactly one character not in the set.
[!… ] When ! is the first character, matches exactly one character not in the set.
[x-x] Matches exactly one of a range of characters.
[:xxxxx:] For details about this type of wildcard, see any POSIX-standard guide to globbing.

Globbing Extensions: Wildcards and special characters

no / or * at end of pattern Matches files only.
/ at end of pattern Matches directories only. With -N, no files under matched directories or their subdirectories are included in the transfer. All subdirectories are still included, although their files are not included. However, with -E, excluding a directory also excludes all files and subdirectories under it.
* or /** at end of pattern Matches both directories and files.
/** Like * but also matches / and the . in /..
/ at start of pattern Must match the entire string from the root of the transfer set. (Note: The leading / does not refer to the system root or the docroot.)

Standard globbing examples

Wildcard Example Matches Does Not Match
/ abc/def/xyz abc/def/xyz abc/def
\ abc\? abc? abc\? abc/D abcD
* abc*f abcdef abc.f abc/f abcefg
? abc?? abcde abc.z abcdef abc/d abc/.
[ … ] [abc]def adef cdef abcdef ade
[^… ] [^abc]def zdef .def 2def bdef /def /.def
[!… ] [!abc]def zdef .def 2def cdef /def /.def
[:xxxxx:] [[:lower:]]def cdef ydef Adef 2def .def

Globbing extension examples

Wildcard Example Matches Does Not Match
/** a/**/f a/f a/.z/f a/d/e/f a/d/f/ za/d/f
* at end of rule abc* abc/ abcfile
/** at end of rule abc/** abc/.file abc/d/e/ abc/
/ at end of rule abc/*/ abc/dir abc/file
no / at end of rule abc abc (file) abc/
/ at start of rule /abc/def /abc/def xyz/abc/def

Testing your filter rules

You can use this procedure to test your filtering rules.

  1. On your computer, create a set of directories and files (size can be small) that approximate a typical transfer file set. In the following example, the file set is in /tmp/src.
  2. Upload the file set to a server. For example, 
    # ascp /tmp/src my_user_name@my_demo.example.com:Upload/

    Where the user is my_user_name, and the target is the Upload directory.

    At the prompt, enter my_user_name user password.

  3. Create a destination directory on your computer, for example, /tmp/dest.
  4. Download your files from the demo server to /tmp/dest to test your filtering rules. For example, 
    # ascp -N 'wxy/**' -E 'def' my_user_name@my_demo.example.com:Upload/src/ /tmp/dest
  5. Compare the destination directory with the source to determine whether the filter behaved as expected. 
    $ diff -r dest/ src/

    The diff output shows the missing files and directories (those that were not transferred).

Example filter rules

The example rules are based on running a command such as the following to download a directory AAA from my_demo.example.com to /tmp/dest:

# ascp rules aspera@my_demo.example.com:Upload/AAA /tmp/dest

The examples use the following file set:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile

Key for interpreting example results:

< xxx/yyy = Excluded
xxx/yyy = Included
zzz/ = directory name
zzz = file name

  1. Transfer everything except files and directories that starts with ".": 
    -N '*' -E 'AAA/**'

    Results:

    AAA/abc/def
AAA/abc/wxy/def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/.def
  2. Exclude directories and files whose names start with wxy: 
    -E 'wxy*'

    Results:

    AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile
  3. Include directories and files that start with wxy if they fall directly under AAA: 
    -N 'wxy*' -E 'AAA/**'

    Results:

    AAA/wxy/
AAA/wxyfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxy/xyx/
< AAA/wxy/xyxfile
  4. Include directories and files at any level that start with wxy, but do not include dot-files, dot-directories, or any files under the wxy directories (unless they start with wxy). However, subdirectories under wxy are included: 
    -N '*/wxy*' -E 'AAA/**'

    Results:

    AAA/abc/wxy/tuv/
AAA/abc/xyz/def/wxy
AAA/wxyfile 
AAA/wxy/xyx/
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def     * 
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/wxy/xyxfile

    * Even though wxy is included, def is excluded because it is a file.

  5. Include wxy directories and files at any level, even those that start with ".": 
    -N '*/wxy*' -N '*/wxy/**' -E 'AAA/**'

    Results:

    AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
  6. Exclude directories and files that start with wxy, but only those found at a specific location in the tree: 
    -E '/AAA/abc/wxy*'

    Results:

    AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
  7. Include the wxy directory at a specific location, and include all its subdirectories and files, including those starting with ".": 
    -N 'AAA/abc/wxy/**' -E 'AAA/**'

    Results:

    AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile
Surround patterns that contain wildcards with single quotation marks (' ') to prevent filter.