Composing a Sync session

Sync has more than 80 options that can be used when composing an async session, but only a few are required, and for security by using several others.

For a complete list and descriptions of available options, see the Async command reference. For configuration and option usage that is required to synchronize with AWS S3 storage, see Synchronizing with AWS S3 Storage.

Note: These instructions describe how to compose a bidirectional async session between a Windows client and a Linux server. They include the required and the recommended options in the correct order. You can use the short form or long form (POSIX) option tags. The complete set of commands by using both tag formats are summarized at the end of the instructions.
  1. Confirm that both endpoints have Sync-enabled licenses and that the remote endpoint is running an Aspera transfer server application (HSTS or HSTE).
    Run ascp -A in the command line and look for sync2 in the Enabled settings section.
  2. Begin by starting async.
    > async
  3. Enter instance options.
    Instance options are used to configure the local (client) computer for the async session and are mostly optional. Include -L log_dir (or --alt-logdir=log_dir) to set client-side logging to a directory that you can access because you might not have permission to access the log in its default location (see Logging). The logging directory must not be in the directory that is being synchronized.
    For example, if the Windows client's username is morgana, morgana can use -L to log to a directory in the home folder:
    async -L "C:\Users\morgana\Aspera jobs\log"

    In this example, the path must be in quotation marks because the path includes a folder name that contains a space. For more information about path formatting, see Async command reference.

  4. Name the session by using the -N option (or --name=pair).
    -N pair is required in async commands. The value for pair is a name that uniquely identifies the Aspera Sync session and is visible in IBM Aspera Console. -N pair must follow any instance options and must precede all session arguments. Names can use only standard alphanumeric characters, plus "_" and "-" characters.
    Note: If your remote host is an Aspera cluster, ensure that your session name is unique by naming the session with a descriptive string followed by the UUID of the local host, such as "cluster-sync-ba209999-0c6c-11d2-97cf-00c04f8eea45".

    For example, name the session job1:

    async -L "C:\Users\morgana\Aspera jobs\log" -N job1

    Once you name the session, you enter the session options. Session options define the transfer parameters including authentication, transfer rate and policy, database storage, and the folders to synchronize.

  5. Provide authentication credentials.
    Sync supports three methods of authenticating to the server: SSH key, password, and basic token. Use SSH keys, unless your server requires a basic token.
    • SSH key: To use SSH key authentication, your SSH public key must be configured on the remote server. For instructions on creating keys and setting them up on the server, see the IBM Aspera High-Speed Transfer Server. Specify the path to your private key file by using the -i file (or --private-key-path=file) option.
    • Password: The password is the one associated with the Aspera transfer user account on the server. You can provide the password as an environment variable (ASPERA_SCP_PASS) or when prompted after starting the command.
    • Basic token: Basic tokens are used for synchronizing to Aspera products that require access key authentication, such as IBM Aspera on Cloud transfer service (AoCts) or IBM Aspera Transfer Cluster Manager (ATC Manager). For instructions on creating the basic token, see Sync with basic token authorization. You can provide the token as an environment variable (ASPERA_SCP_TOKEN) or in the command line by using the -W token_string (or --token=token_string) option.

    For example, use -i and specify the path to morgana's SSH private key in their home folder:

    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa

    In this case, the path to the SSH key can use platform independent of path separators (/) and be entered without quotation marks around it because it does not have a space in it.

  6. If the local data is stored on a mount or object storage, specify the locations for the local snapshot database.
    The snapshot database cannot be located on CIFS, NFS, or other shared file systems mounted on Linux. If the local files and directories that are specified in the previous step are on a mount, you must specify a local location by using -b db_dir (or --local-db-dir=db_dir). The database must not be in the directory that is being synchronized.

    For example, use -b to store the local snapshot database in morgana's "Aspera jobs" folder:

    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db"
  7. Set transfer parameters.
    The same transfer rate and transfer policy options that are used to control ascp transfers can be applied to async sessions. Set a target rate that is based on your available bandwidth and system capabilities. Set the target (maximum) rate by using -l rate (or --target-rate=rate).
    For example, use -l to set the target rate to 500 Mbps:
    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db" -l 500m
  8. Specify the local directory for synchronization.
    Enter the local directory by using -d ldir (or --local-dir=ldir).

    For example, use -d to set the local directory to morgana's data folder:

    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db" -l 500m -d c:/users/morgana/data
  9. Specify the transfer username, remote host, and remote directory for synchronization.
    Unlike previous options for which one short option flag was equivalent to one long option flag, when specifying the username, remote host, and remote directory, the short flag option is the equivalent of one to three long option flags. For example, if the username is morgana, the remote host IP address is 10.0.0.1, and the remote directory is /data, then the following options are equivalent to each other:
    -r morgana@10.0.0.1:/data
    --remote-dir=morgana@10.0.0.1:/data
    --user=morgana --remote-dir=10.0.0.1:/data
    --user=morgana --host=10.0.0.1 --remote-dir=/data

    If the name of your remote directory contains an "@", use the --user option so that the "@" is not treated specially in the argument for --remote-dir.

    For example, use -r to set the username, remote host, and remote directory:
    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db" -l 500m -d c:/users/morgana/data -r morgana@10.0.0.1:/data
  10. If a source directory is on an NFS or CIFS mount, require Sync to use the mount signature file.
    Warning: If you do not use the mount signature file and the NFS or CIFS mount is unreachable, Sync considers those files as deleted and deletes them from the other endpoint.
    If the local endpoint is on an NFS or CIFS mount and the Sync is push or bidirectional, use --local-mount-signature. If the remote endpoint is on an NFS or CIFS mount and the Sync is pull or bidirectional, use --remote-mount-signature.
  11. Specify the locations for the remote Sync log and database.
    On the server, Sync logs to the default location (see Logging) if no location is specified for <async_log_dir> in the server's configuration file. Use -R (or --remote-logdir) to specify a logging location to which you have access. The location must be within your docroot on the server, unless you are synchronizing with AWS S3 object storage. -R is overridden by the server's configuration file. If you are restricted to aspshell on the server, you cannot use this option.

    Also, you can use -B (or --remote-db-dir) to specify a location for the remote Sync database. As with the log file, the location must be within your docroot, it is overridden by <async_db_dir> in the server's configuration file, and you cannot use this option if you are restricted to aspshell.

    As on the local computer, the Sync log and database must not be in a directory that is being synchronized.

    For example, to set the remote log and snapshot database files to morgana's home folder:
    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db" -l 500m -d c:/users/morgana/data -r morgana@10.0.0.1:/data -R /morgana/async/log -B /morgana/async/db
  12. Specify the synchronization mode.
    Sync can be run in three modes:
    • push: The contents of ldir are synchronized to rdir, with the ldir content that overwrites the rdir content, by default (unless the overwrite options are specified otherwise, such as to overwrite only if rdir is older, or never overwrite).
    • pull: The contents of rdir are synchronized to ldir, with the rdir content that overwrites the ldir content, by default.
    • bidi (bidirectional): The contents of ldir and rdir are synchronized, with newer versions of files and directories that overwrite older versions in either ldir or rdir, by default.
    To synchronize the remote folder with the local folder use -K push (or --direction=push).
    For example, use -K bidi to do a bidirectional sync:
    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db" -l 500m -d c:/users/morgana/data -r morgana@10.0.0.1:/data -R /morgana/async/log -B /morgana/async/db -K bidi
  13. Preserve file attributes.
    When a file or directory is transferred between computers, the file is written to the destination as the transfer user and the file modification time (and creation time on Windows) are reset. Most users prefer to preserve timestamps from the source to the destination by using the -t option.
    For example, use -t timestamps:
    async -L "C:\Users\morgana\Aspera jobs\log" -N job1 -i c:/users/morgana/.ssh/id_rsa -b "C:\Users\morgana\Aspera jobs\db" -l 500m -d c:/users/morgana/data -r morgana@10.0.0.1:/data -R /morgana/async/log -B /morgana/async/db -K bidi -t
    Note: When synchronizing between Unix-like operating systems, you can also preserve the user IDs (uid) and group IDs (gid) from the source to the destination by using the options -u -j (equivalent to --preserve-uid --preserve-gid).

    Extended file attributes and ACLs can also be preserved; see the Async command reference. When using --dedup, file metadata preservation is supported for copy.

Summary

The instructions created the following Sync session, which is shown by using short option flags and POSIX (long) flags. Each option is shown on a separate line for clarity, but must be entered in the command line as a single line.

Warning: This example does not include the option to make Sync check for a mount signature file. If a source is on an NFS or CIFS mount, include --local-mount-signature and --remote-mount-signature to prevent Sync from deleting files on an endpoint if a mount becomes unavailable. For instructions, see Configuring Sync endpoints.

Using short-format option flags:

async 
    -L "C:\Users\morgana\Aspera jobs\log" 
    -N job1 
    -i c:/users/morgana/.ssh/id_rsa 
    -b "C:\Users\morgana\Aspera jobs\db" 
    -l 500m 
    -d c:/users/morgana/data 
    -r morgana@10.0.0.1:/data
    -R /morgana/async/log 
    -B /morgana/async/db 
    -K bidi 
    -t

Using long-format option flags:

async 
    --alt-logdir="C:\Users\morgana\Aspera jobs\log" 
    --name=job1 
    --private-key-path=c:/users/morgana/.ssh/id_rsa 
    --local-db-dir="C:\Users\morgana\Aspera jobs\db" 
    --target-rate=500m 
    --local-dir=c:/users/morgana/data 
    --user=morgana
    --host=10.0.0.1
    --remote-dir=/data
    --remote-logdir=/morgana/async/log 
    --remote-db-dir=/morgana/async/db  
    --direction=bidi 
    --preserve-time

If the session is between Linux computers, it also includes the following session options:

-u
-j

Or using long-format option flags:

--preserve-uid
--preserve-gid