Reporting checksums
File checksums are useful for troubleshooting file corruption, allowing to determine at what point in the transfer file corruption occurred. Aspera servers can report source file checksums that are calculated in real time during transfer and then sent from the source to the destination.
To support checksum reporting, the transfer must meet both of the following requirements:
- Both the server and client computers must be running HSTS or HSTE.
- The transfer must be encrypted. Encryption is enabled by default.
The user on the destination can calculate a checksum for the received file and compare it (manually or programmatically) to the checksum reported by the sender. The checksum reported by the source can be retrieved in the destination logs, a manifest file, in IBM Aspera Console, or as an environment variable. Instructions for comparing checksums follow the instructions for enabling checksum reporting.
Checksum reporting is disabled by default. Enable and configure checksum reporting on the server by using the following methods:
- Edit aspera.conf with asconfigurator.
- Set options in the client GUI.
- Set ascp command line options (per-transfer configuration).
Command line options override the settings in aspera.conf and the GUI.
Overview of checksum configuration options
asconfigurator OptionGUI Setting ascp Option |
Description |
---|---|
file_checksum File checksum method --file-checksum=type |
Enable checksum reporting and specify the type of checksum to calculate for
transferred files.
Note: The default value for the ascp option is
none , in which case the
reported checksum is the one configured on the server, if any. |
file_manifest File Manifest --file_manifest=output |
The file manifest is a file that contains a list of content that was
transferred in a transfer session. The file name of the file manifest is
automatically generated from the transfer session ID. When set to
When set to |
file_manifest_path File Manifest Path --file_manifest_path=path |
The location where manifest files are written. The location can be an absolute path or
a path relative to the transfer user's home directory. If no path is specified (default), the file
is generated under the destination path at the receiver, and under the first source path at the
sender. Note: File manifests can be stored only locally. Thus, if you are using S3 or other nonlocal
storage, you must specify a local manifest path.
|
Enabling checksum reporting by editing aspera.conf
To enable checksum reporting, run the following command:
> asconfigurator -x "set_node_data;file_checksum,checksum"
To enable and configure the file manifest where checksum report data is stored, run the following commands:
> asconfigurator -x "set_node_data;file_manifest,text"
> asconfigurator -x "set_node_data;file_manifest_path,filepath"
These commands create lines in aspera.conf as shown in the following example, where checksum type is md5, file manifest is enabled, and the path is C:\Users\Public\reports.
<file_system>
...
<file_checksum>md5</file_checksum>
<file_manifest>text</file_manifest>
<file_manifest_path>C:\Users\Public\reports
</file_manifest_path>
...
</file_system>
Enabling checksum reporting from the GUI
Click Configuration to open the Server Configuration window. Select the Global, Groups, or Users tab, depending on whether you want to enable checksum reporting for all users, or for a particular group or user.
Under the File Handling tab, locate the setting for File
checksum method. Check the override box and for the effective value, select
any
, md5
, sha1
, sha256
,
sha384
, or sha512
.
To enable the file manifest, select the override checkbox for File Manifest and set the effective value to text. To set the path, select the override checkbox for File Manifest Path and set the effective value to the folder in which you want the manifest files saved.
In the examples, the manifest is generated when files are transferred and saved as a text file called aspera-transfer-transfer_id-manifest.txt in the directory C:\Users\Public\reports.
Enabling checksum reporting in an ascp session
To enable checksum reporting on a per-transfer-session basis, run ascp with the
--file-checksum=hash option, where
hash is sha1
, md5
,
sha-512
, sha-384
, sha-256
, or
none
(the default).
Enable the manifest with
--file-manifest=output where
output is either text
or
none
. Set the path to the manifest file with
--file-manifest-path=path.
For example,
> ascp --file-checksum=md5 --file-manifest=text --file-manifest-path=C:\Users\Public\reports
file aspera_user_1@189.0.202.39:/destination_path
Setting up a processing script
An alternative to enabling and configuring the file manifest to collect checksum reporting is to set up a script to report the values. See Automated execution of Lua scripts with transfer Events.
Comparing checksums
If you open a file that you downloaded with Aspera and find that it is corrupted, you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source.
- Retrieve the checksum that was calculated by Aspera as the file was
transferred.
- If you specified a file manifest and file manifest path as part of an ascp transfer or Lua transfer event script, the checksums are in that file in the specified location.
- If you specified a file manifest and file manifest path in the GUI or aspera.conf, the checksums are in a file that is named aspera-transfer-transfer_id-manifest.txt in the specified location.
- Calculate the checksum of the corrupted file. This example uses the MD5 checksum method;
replace MD5 with the appropriate checksum method if you use a different
one.
> CertUtil -hashfile filepath MD5
- Compare the checksum reported by Aspera with the checksum that you calculated
for the corrupted file.
- If they do not match, then corruption occurred as the file was written to the destination. Download the file again and confirm that it is not corrupted. If it is corrupted, compare the checksums again. If they do not match, investigate the write process or attempt another download. If they match, continue to the next step.
- If they match, then corruption might have occurred as the file was read from the source. Continue to the next step.
- Calculate the checksums for the file on the source. These examples use the MD5
checksum method; replace MD5 with the appropriate checksum method if you use a
different
one.Windows:
> CertUtil -hashfile filepath MD5
Mac OS X:$ md5 filepath
Linux and Linux on z Systems:# md5sum filepath
AIX:# csum -h MD5 filepath
Solaris:# digest -a md5 -v filepath
- Compare the checksum of the file on the source with the one reported by Aspera.
- If they do not match, then corruption occurred when the file was read from the source. Download the file again and confirm that it is not corrupted on the destination. If it is corrupted, continue to the next step.
- If they match, confirm that the source file is not corrupted. If the source file is corrupted, replace it with an uncorrupted one, if possible, and then download the file again.