Configuring filters to include and exclude files

For example, on Windows FAT or NTFS (or directories) to transfer by indicating which to skip or include based on name matching. When no filtering rules are specified by the client, ascp transfers all source files in the transfer list.

Filters can be specified on the ascp command line and in aspera.conf. Ascp applies filtering rules that are set in aspera.conf before it applies rules on the command line.

Set filtering rules in the GUI

  1. Click Configuration > File Handling.
    File Handling tab, enable File Filter Pattern list
  2. Scroll down to File Filter Pattern List.
  3. Select Override then click Filter define button to open the filter Define dialog. If rules were added earlier, either through the GUI or through aspera.conf, they appear in the window.
    Click Plus to define filtering rules
  4. Click plus sign in the top left corner to add a filtering rule, or click minus sign in top left corner to delete a rule that you selected.

Rule syntax

A rule consists of a + or - sign that indicates whether to include or exclude a rule, followed by a space character and by a pattern. A pattern can be a file or directory name, or a set of names expressed with UNIX glob patterns.

Note: Do not confuse the GUI line add and line delete buttons in the GUI: plus sign in the top left corner and minus sign in top left corner, with the include or exclude characters + or - that are part of rule syntax. The purpose of each is different and they are unrelated.

Basic usage

  • Filtering rules are applied to the transfer list in the order from top to bottom.
  • Filtering is a process of exclusion, and the include rules overrides the exclude rules that follow them. Include rules cannot add back files that are excluded by a preceding exclude rule.
  • Include rules must be followed by at least one exclude rule, otherwise all files are transferred because none are excluded. To exclude all unmatched files, add two final rules: "- *" and "- .*".
  • Filtering operates only on the set of files and directories in the transfer list. An include rule cannot add files or directories that are not already part of the transfer list.
Example Transfer Result
- rule Transfer all files and directories except those with names that match rule.
+ rule Transfer all files and directories because none are excluded.

To transfer only the files and directories with names that match rule use:

+ rule
- *
- .*
+rule1
- rule2
Transfer all files and directories with names that match rule1, as well as all other files and directories except those with names that match rule2.
- rule1
+ rule2
Transfer all files and directories except those with names that match rule1. All files and directories that are not already excluded by rule1, are included because no additional exclude rule follows -N rule2. Additional filters can be set for transfers in the UI (Adding and editing connections) or with the command line (Using filters to include and exclude files).

To transfer only the files and directories with names that do not match rule1 but do match rule2 use:

- rule1
+ rule2
- *
- .*

Filtering rule application

Filtering order

Filtering rules are applied to the transfer list in the order that they appear in the list.

  1. The first file or directory in the transfer list is compared to the pattern of the first rule.
  2. If the file matches the pattern, Ascp includes it or excludes it and the file is immune to any following rules.
    Note: When a directory is excluded, directories and files in it are also excluded and are not compared to any following rules.
  3. If the file does not match, it is compared to the next rule and repeats the process for each rule until a match is found or until all rules have been tried.
  4. If the file never matches any exclude rules, it is included in the transfer.
  5. The next file or directory in the transfer list is then compared to the filtering rules until all eligible files are evaluated.

Example

Consider the following set of rules:

+ file2
- file[0-9]

If the source contains file1, file2, and fileA, the filtering rules are applied as follows:

  1. file1 is compared with the first rule (+ file2) and does not match so filtering continues.
  2. file1 is compared with the second rule (- file[0-9]) and matches, so it is excluded from the transfer.
  3. file2 is compared with the first rule and matches, so it is included in the transfer and filtering stops for file2.
  4. fileA is compared with the first rule and does not match so filtering continues.
  5. fileA is compared with the second rule and does not match. Because no rules exclude it, fileA is included in the transfer.

Rule patterns

Rule patterns (globs) use standard globbing syntax that includes wildcards and special characters, and several Aspera extensions to the standard.

  • Character case: Case always matters, even if the file system does not enforce such a distinction. For example, on Windows FAT or NTFS file systems and macOS HPFS+ a file system search for DEBUG returns files Debug and debug. In contrast, ascp filter rules use exact comparison, such that debug does not match Debug. To match both, use [Dd]ebug.
  • Partial matches: With globs, unlike standard regular expressions, the entire file name or directory name must match the pattern. For example, the pattern abc*f matches abcdef but not abcdefg.

Standard globbing: Wildcards and special characters

/ The only recognized path separator.
\ Quotes any character literally, including itself. \ is exclusively a quoting operator, not a path separator.
* Matches zero or more characters, except / or the . in /..
? Matches any single character, except / or the . in /..
[] Matches exactly one of a set of characters, except / or the . in /..
[^] When ^ is the first character, matches exactly one character not in the set.
[!] When ! is the first character, matches exactly one character not in the set.
[x-x] Matches exactly one of a range of characters.
[:xxxxx:] For details about this type of wildcard, see any POSIX-standard guide to globbing.

Globbing Extensions: Wildcards and special characters

no / or * at end of pattern Matches files only.
/ at end of pattern Matches directories only. With -N, no files under matched directories or their subdirectories are included in the transfer. All subdirectories are still included, although their files are not included. However, with -E, excluding a directory also excludes all files and subdirectories under it.
* or /** at end of pattern Matches both directories and files.
/** Like * but also matches / and the . in /..
/ at start of pattern Must match the entire string from the root of the transfer set. (Note: The leading / does not refer to the system root or the docroot.)

Standard globbing examples

Wildcard Example Matches Does Not Match
/ abc/def/xyz abc/def/xyz abc/def
\ abc\? abc? abc\? abc/D abcD
* abc*f abcdef abc.f abc/f abcefg
? abc?? abcde abc.z abcdef abc/d abc/.
[] [abc]def adef cdef abcdef ade
[^] [^abc]def zdef .def 2def bdef /def /.def
[!] [!abc]def zdef .def 2def cdef /def /.def
[:xxxxx:] [[:lower:]]def cdef ydef Adef 2def .def

Globbing extension examples

Wildcard Example Matches Does Not Match
/** a/**/f a/f a/.z/f a/d/e/f a/d/f/ za/d/f
* at end of rule abc* abc/ abcfile  
/** at end of rule abc/** abc/.file abc/d/e/ abc/
/ at end of rule abc/*/ abc/dir abc/file
no / at end of rule abc abc (file) abc/
/ at start of rule /abc/def /abc/def xyz/abc/def

Testing your filtering rules

You can use this procedure to test your filtering rules.

  1. On your computer, create a small set of directories and files that generally matches a file set you typically transfer. Since filenames are all that matter, the size of the files can be small.
  2. Place the file set in an accessible location, for example C:\tmp\src.
  3. Upload the file set to your server. For information about setting up a connection, see Testing a locally initiated transfer.
  4. Create a destination directory on your computer, for example C:\tmp\dest.
  5. You can now download your files from the server to C:\tmp\dest and test your filtering rules. For example:
    Testing your filtering rules
  6. Compare the list of files transferred to the list of your original files.

Example filter rules

These example rules are based on downloading a directory AAA from your server to C:\tmp\dest on your computer:

Example filter rules

These examples use the following file set:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile

Key for interpreting results:

< xxx/yyy = Excluded
xxx/yyy = Included
zzz/ = directory name
zzz = filename

1- Transfer everything except files and directories that start with ".":

+ * 
- AAA/**

Results:

AAA/abc/def
AAA/abc/wxy/def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/.def

2- Exclude directories and files whose names start with wxy

- wxy* 

Results:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile

3- Include directories and files that start with wxy if they fall directly under AAA:

+ wxy* 
- AAA/**

Results:

AAA/wxy/
AAA/wxyfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/abc/xyz/def/wxy
< AAA/wxy/xyx/
< AAA/wxy/xyxfile

4- Include directories and files at any level that start with wxy, but do not include dot-files, dot-directories, or any files under the wxy directories, unless they start with wxy. However, subdirectories under wxy are included:

+ */wxy*
- AAA/** 

Results:

AAA/abc/wxy/tuv/
AAA/abc/xyz/def/wxy
AAA/wxyfile 
AAA/wxy/xyx/
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/wxy/def     * 
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def
< AAA/wxy/xyxfile

* Even though wxy is included, def is excluded because it's a file.

5- Include wxy directories and files at any level, even those starting with ".":

+ */wxy*
- */wxy/**
- AAA/**

Results:

AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def

6- Exclude directories and files that start with wxy, but only those found at a specific location in the tree:

+ /AAA/abc/wxy* 

Results:

AAA/abc/def
AAA/abc/.def
AAA/abc/.wxy/def
AAA/abc/xyz/def/wxy
AAA/wxyfile
AAA/wxy/xyx/
AAA/wxy/xyxfile
< AAA/abc/wxy/def
< AAA/abc/wxy/.def
< AAA/abc/wxy/tuv/def

7- Include the wxy directory at a specific location, and include all its subdirectories and files, including those starting with ".":

+ AAA/abc/wxy/**
- AAA/** 

Results:

AAA/abc/wxy/def
AAA/abc/wxy/.def
AAA/abc/wxy/tuv/def
< AAA/abc/def
< AAA/abc/.def
< AAA/abc/.wxy/def
< AAA/abc/xyz/def/wxy
< AAA/wxyfile
< AAA/wxy/xyx/
< AAA/wxy/xyxfile