Reporting checksums

File checksums are useful for troubleshooting file corruption, allowing to determine at what point in the transfer file corruption occurred. Aspera servers can report source file checksums that are calculated in real time during transfer and then sent from the source to the destination.

To support checksum reporting, the transfer must meet both of the following requirements:

  • Both the server and client computers must be running HSTS or HSTE.
  • The transfer must be encrypted. Encryption is enabled by default.

The user on the destination can calculate a checksum for the received file and compare it (manually or programmatically) to the checksum reported by the sender. The checksum reported by the source can be retrieved in the destination logs, a manifest file, in IBM Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.

Checksum reporting is disabled by default. Enable and configure checksum reporting on the server by using the following methods:

  • Edit aspera.conf with asconfigurator.
  • Set options in the client GUI.
  • Set ascp command line options (per-transfer configuration).

Command line options override the settings in aspera.conf and the GUI.

Important: When checksum reporting is enabled, transfers of very large files (>TB) take a long time to resume because the entire file must be reread.

Overview of checksum configuration options

asconfigurator Option
GUI Setting
ascp Option
Description
file_checksum
File checksum method
--file-checksum=type
Enable checksum reporting and specify the type of checksum to calculate for transferred files.
  • any - Allow the checksum format to be whichever format the client requests. (Default in aspera.conf and the GUI)
  • md5 - Calculate and report the MD5 checksum.
  • sha1 - Calculate and report the SHA-1 checksum.
  • sha256 - Calculate and report the SHA-256 checksum.
  • sha384 - Calculate and report the SHA-384 checksum.
  • sha512 - Calculate and report the SHA-512 checksum.
Note: The default value for the ascp option is none, in which case the reported checksum is the one configured on the server, if any.
file_manifest
File Manifest
--file_manifest=output
The file manifest is a file that contains a list of content that was transferred in a transfer session. The file name of the file manifest is automatically generated from the transfer session ID.

When set to none, no file manifest is created. (Default)

When set to text, a text file is generated that lists all files in each transfer session.

file_manifest_path
File Manifest Path
--file_manifest_path=path
The location where manifest files are written. The location can be an absolute path or a path relative to the transfer user's home directory. If no path is specified (default), the file is generated under the destination path at the receiver, and under the first source path at the sender.
Note: File manifests can be stored only locally. Thus, if you are using S3 or other nonlocal storage, you must specify a local manifest path.

Enabling checksum reporting by editing aspera.conf

To enable checksum reporting, run the following command:

> asconfigurator -x "set_node_data;file_checksum,checksum"

To enable and configure the file manifest where checksum report data is stored, run the following commands:

> asconfigurator -x "set_node_data;file_manifest,text"
> asconfigurator -x "set_node_data;file_manifest_path,filepath"

These commands create lines in aspera.conf as shown in the following example, where checksum type is md5, file manifest is enabled, and the path is C:\Users\Public\reports.

<file_system>
    ...
    <file_checksum>md5</file_checksum>
    <file_manifest>text</file_manifest>
    <file_manifest_path>C:\Users\Public\reports</file_manifest_path>
    ...
</file_system>

Enabling checksum reporting from the GUI

Click Configuration to open the Server Configuration window. Select the Global, Groups, or Users tab, depending on whether you want to enable checksum reporting for all users, or for a particular group or user.

Under the File Handling tab, locate the setting for File checksum method. Check the override box and for the effective value, select any, md5, sha1, sha256, sha384, or sha512.

Enabling checksum reporting from the GUI

To enable the file manifest, select the override checkbox for File Manifest and set the effective value to text. To set the path, select the override checkbox for File Manifest Path and set the effective value to the folder in which you want the manifest files saved.

Enabling manifest reporting from the GUI

In the examples, the manifest is generated when files are transferred and saved as a text file called aspera-transfer-transfer_id-manifest.txt in the directory C:\Users\Public\reports.

Enabling checksum reporting in an ascp session

To enable checksum reporting on a per-transfer-session basis, run ascp with the --file-checksum=hash option, where hash is sha1, md5, sha-512, sha-384, sha-256, or none (the default).

Enable the manifest with --file-manifest=output where output is either text or none. Set the path to the manifest file with --file-manifest-path=path.

For example,

> ascp --file-checksum=md5 --file-manifest=text --file-manifest-path=C:\Users\Public\reports file aspera_user_1@189.0.202.39:/destination_path

Setting up a processing script

An alternative to enabling and configuring the file manifest to collect checksum reporting is to set up a script to report the values. See Automated execution of Lua scripts with transfer Events.

Comparing checksums

If you open a file that you downloaded with Aspera and find that it is corrupted, you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source.

  1. Retrieve the checksum that was calculated by Aspera as the file was transferred.
    • If you specified a file manifest and file manifest path as part of an ascp transfer or Lua transfer event script, the checksums are in that file in the specified location.
    • If you specified a file manifest and file manifest path in the GUI or aspera.conf, the checksums are in a file that is named aspera-transfer-transfer_id-manifest.txt in the specified location.
  2. Calculate the checksum of the corrupted file. This example uses the MD5 checksum method; replace MD5 with the appropriate checksum method if you use a different one.
    > CertUtil -hashfile filepath MD5
  3. Compare the checksum reported by Aspera with the checksum that you calculated for the corrupted file.
    • If they do not match, then corruption occurred as the file was written to the destination. Download the file again and confirm that it is not corrupted. If it is corrupted, compare the checksums again. If they do not match, investigate the write process or attempt another download. If they match, continue to the next step.
    • If they match, then corruption might have occurred as the file was read from the source. Continue to the next step.
  4. Calculate the checksums for the file on the source. These examples use the MD5 checksum method; replace MD5 with the appropriate checksum method if you use a different one.
    Windows:
    > CertUtil -hashfile filepath MD5
    Mac OS X:
    $ md5 filepath
    Linux and Linux on z Systems:
    # md5sum filepath
    AIX:
    # csum -h MD5 filepath
    Solaris:
    # digest -a md5 -v filepath
  5. Compare the checksum of the file on the source with the one reported by Aspera.
    • If they do not match, then corruption occurred when the file was read from the source. Download the file again and confirm that it is not corrupted on the destination. If it is corrupted, continue to the next step.
    • If they match, confirm that the source file is not corrupted. If the source file is corrupted, replace it with an uncorrupted one, if possible, and then download the file again.