Using High-Speed Add-on (HSAO) for Connect:Direct

IBM Aspera High-Speed Add-on for Connect:Direct® for UNIX uses FASP (Fast and Secure Protocol) network transport to transfer files over high bandwidth and high latency network connections.

At low latency it performs similarly to TCP/IP. However, as latency and packet loss increase, unlike TCP/IP, its performance does not degrade, and FASP continues to take advantage of all the available bandwidth.

IBM Aspera High-Speed Add-on for Connect:Direct for UNIX supports interoperability with Connect:Direct for Microsoft Windows and Secure Proxy. Fore more information, refer to High-Speed Add-On Getting Started Guide.

Note: Secure+ is used to secure FASP transfers exactly the same way it is used for TCP/IP transfers.

Activating FASP

By default, IBM Aspera High-Speed Add-on for Connect:Direct is not enabled. To enable it, you must download a license key and install Connect:Direct for UNIX.

About this task

Procedure

  1. Download the IBM Aspera High-Speed Add-on for Connect:Direct license key for your Connect:Direct node from Passport Advantage.
  2. Rename the file aspera-license.
  3. Save the renamed file to the <cd_dir>/ndm/cfg/<nodename> directory.
  4. Download and install Connect:Direct for UNIX from Fix Central.
    Important: The Connect:Direct install package includes the IBM Aspera High-Speed Add-on for Connect:Direct configuration file (aspera.conf). It contains the minimum necessary basic configuration statements to use FASP on Connect:Direct. It is always installed even if you do not purchase IBM Aspera High-Speed Add-on for Connect:Direct. Do NOT make any changes to this file.

Licensed bandwidth for FASP transactions

The bandwidth available to a file transfer is limited by, among other things, the bandwidths specified in the sender’s and receiver’s Aspera license keys.

There are two types of available license keys:
  • Datacenter licenses (available in 10gbps, 1gbps, 300mbps and 100mbps) - can send and receive files using FASP when connected to a node that has an Endpoint or DataCenter license.
  • Endpoint license - can send and receive files using FASP when connected to a node that has a DataCenter license.
Note: Aspera FASP bandwidth setting should not exceed the network capability. As per the available network bandwidth in your network, start with a reasonable setting e.g. 100 mbps. FASP bandwidth setting is likely to max out the resources especially CPU. FASP requires more CPU than normal TCP/IP transfer and secure+ requires even more CPU than Non-Secure+ transfer. In case of high CPU usage (reaching approx. 100%), either add more CPU resources or reduce the FASP bandwidth setting.

When both sender and receiver only have Endpoint licenses, file transfer over FASP is not supported. When either the sender or receiver has an Endpoint license and the other has a Datacenter license, the available bandwidth is limited to the value in the Datacenter license. When both sender and receiver have Datacenter licenses, the bandwidth is limited to the smaller of the two values in the Datacenter licenses.

FASP Process Language

Once the FASP parameters for both trading partners have been configured, you can override the default settings on a process by process basis to perform exception processing.

Optional Parameters

FASP Parameters:
  • FASP (Yes | No)
  • FASP POLICY (Values are the same as the FASP Local and Remote node record parameters)
  • FASP.FILESIZE.THRESHOLD (Values are the same as the FASP Local and Remote node record parameters)
  • FASP.BANDWIDTH (Values are the same as the FASP Local and Remote node record parameters)
FASP Parameters are applicable in three different contexts:
  • COPY statement - The four FASP parameters may be used individually or as a group within a COPY statement. This will set FASP values for the duration of that COPY statement and will not have any effect on statements within the submitted Process
  • PROCESS statement - The four FASP parameters may be used individually or as a group at the end of a PROCESS statement. This will set the FASP parameters for all of the COPY statements in the process
  • SUBMIT command - The four FASP parameters may be set individually or as a group at the end of a SUBMIT command. This will set the FASP parameters for all COPY statements in the process being submitted These settings will set FASP information for their relevant part of the scope, potentially overriding the Local Node settings, Remote Node settings and each other.

Examples

Copy statement example:
step01 copy
from
(
file = /tmp/exampleout
pnode
)
ckpt = 2M
compress extended
fasp=yes
fasp.policy=fixed
fasp.bandwidth=500M
fasp.filesize.threshold=10G
to
(
file = /tmp/examplein
snode
disp = rpl
)
Process statement example:
SAMPLE PROCESS    SNODE=WINVM-470
fasp=yes
fasp.policy=fixed
fasp.bandwidth=500M
fasp.filesize.threshold=10G
step01 copy
from
(
file = /tmp/exampleout
pnode
)
ckpt = 2M
compress extended
to
(
file = /tmp/examplein
snode
disp = rpl
)
PEND

Hierarchy Settings

The system uses the following hierarchy to process overrides:
  1. Remote node record overrides local node record.
  2. Process parameters override remote node record.
  3. Submit statement overrides the process parameters.
  4. Each Copy statement overrides the effective settings of the session established by the node settings, Process or Submit statements. The Copy statement override is effective only for the duration of the Copy step.

Using FASP with IBM Aspera High-Speed Add-on and Secure Proxy

You can send files using IBM Aspera High-Speed Add-on through Secure Proxy using Connect:Direct for UNIX.

If you send a file from your local Connect:Direct for UNIX node configured for FASP, it passes through your Secure Proxy instance using FASP, and is sent to the remote node.

In addition to the FASP parameter values outlined in Configuring FASP, the following parameter should be used when using Secure Proxy between Connect:Direct nodes:
fasp=(yes|no|ssp,yes|no|ssp)
The first parameter is the default for Connect:Direct as the PNODE. The second parameter is the default for Connect:Direct as the SNODE.

This parameter can now be used in the netmap local node record and remote node trading partner record in Connect:Direct for UNIX.

The following table shows results when Connect:Direct FASP protocol is used between two Connect:Direct nodes with no Sterling Secure Proxy involved.

PNODE fasp= Protocol SNODE fasp=
N TCP N
N TCP Y
N TCP SSP
Y TCP N
Y C:D FASP Y
Y TCP SSP
SSP TCP N
SSP TCP Y
SSP TCP SSP

The following table shows results when Connect:Direct FASP protocol is used with two Connect:Direct nodes going through a single instance of Sterling Secure Proxy.

PNODE fasp= Protocol SSP Protocol SNODE fasp=
N TCP SSP TCP N
N TCP SSP TCP Y
N TCP SSP TCP SSP
Y TCP SSP TCP N
Y C:D FASP SSP C:D FASP Y
Y C:D FASP SSP TCP SSP
SSP TCP SSP TCP N
SSP TCP SSP C:D FASP Y
SSP TCP SSP TCP SSP

The following table shows results when Connect:Direct FASP protocol is used with two Connect:Direct nodes going through two instances of Sterling Secure Proxy.

PNODE fasp= Protocol SSP Protocol SSP Protocol SNODE fasp=
N TCP SSP TCP SSP TCP N
N TCP SSP TCP SSP TCP Y
N TCP SSP TCP SSP TCP SSP
Y TCP TCP TCP SSP TCP N
Y C:D FASP SSP C:D FASP SSP C:D FASP Y
Y C:D FASP SSP C:D FASP SSP TCP SSP
SSP TCP SSP TCP SSP TCP N
SSP TCP SSP C:D FASP SSP C:D FASP Y
SSP TCP SSP C:D FASP SSP TCP SSP

For more information on using IBM® Secure Proxy with FASP, see Using FASP with Sterling Secure Proxy .

Configuring FASP

About this task

FASP configuration settings are not added to the Connect:Direct configuration files during install. To enable IBM Aspera High-Speed Add-on for Connect:Direct, you must manually configure the initparm.cfg and netmap.cfg files to run FASP.

Procedure

  1. Do one of the following steps:
    • If you installed Connect:Direct for UNIX V4.2.0.4 as a new installation (you did not upgrade from a previous version), go to Step 2. The initparm.cfg file is already configured for FASP listen ports.
    • If you upgraded from a previous version of Connect:Direct for UNIX to V4.2.0.4, you must configure the initparm.cfg file by specifying a FASP listen port or port ranges.

    Format is listen.ports=(nnnnn, nnnnn-nnnnn).

    Example:
    # FASP listen ports
    fasp:\
    :listen.ports=(44001, 33002-33005):
    Note: The number of concurrent FASP processes is limited to the number of ports designated in this file. If you attempt to use more concurrent FASP processes than there are ports available fails, FASP fails.
  2. Configure the netmap.cfg file by specifying FASP values for the local node record. Use the following chart.
    Example:
    local.node:\
    … :fasp=yes:\ :fasp.policy=fair:\ :fasp.bandwidth=500MB:\ :fasp.filesize.threshold=2GB:\
    Parameter Value
    fasp Optional. Default is no if the parameter is not present. Enables FASP.
    • If set to no, FASP is disabled.
    • If set to yes, yes, FASP is enabled. This sets the default for all Connect:Direct file transfers. fasp=(pnode value, snode value), for example, fasp=(yes, ssp)
    • This setting can be overridden by the remote node record or process parameters.
    • The remote server must have FASP enabled.
    fasp.filesize.threshhold Optional. Used to restrict small files from being sent using FASP.
    • If the file is greater than or equal to the stated value, the Connect:Direct server sends the file using FASP. Otherwise, it is sent using TCP/IP.
    • Default is 1GB.
    • You can use KB, MB, or GB designators. If no designator is included, the system uses bits.
    • This setting can be overridden by the remote node record or process parameters.
    fasp.bandwidth Optional. Default is as stipulated in the FASP license key. Specifies how much bandwidth each transfer can use.
    • Default value can be changed, but cannot exceed the bandwidth specified in the license key.
    • You can use KB, MB, or GB designators. If no designator is included, the system uses bits per second.
    • This setting can be overridden by the remote node record or process parameters, but cannot exceed the bandwidth specified in the license key.
    fasp.policy Optional. Specifies the fairness of each transfer. Default is fair.
    • This setting can be overridden by the remote node record or process parameters.
    • Valid values are:
      • Fixed - FASP attempts to transfer at the specified target rate, regardless of the actual network capacity. This policy transfers at a constant rate and finishes in a guaranteed amount of time. This policy typically occupies a majority of the network's bandwidth, and is not recommended in most file transfer scenarios.
      • Fair - FASP monitors the network and adjusts the transfer rate to fully utilize the available bandwidth up to the maximum rate. When other types of traffic build up and congestion occurs, FASP shares bandwidth with other traffic fairly by transferring at an even rate. This is the best option for most file transfer scenarios.
      • High - FASP monitors the network and adjusts the transfer rate to fully utilize the available bandwidth up to the maximum rate. When congestion occurs, a FASP session with high policy transfers at a rate twice of a session with fair policy.
      • Low - Similar to Fair mode, the Low (or Trickle) policy uses the available bandwidth up to the maximum rate as set in the Aspera license file. When congestion occurs, the transfer rate is decreased all the way down to the minimum rate as set in the Aspera license file.
  3. (Optional) Configure the netmap.cfg file by specifying FASP values for the remote node record. Use the following chart. Configure the remote node if you need to override your local node settings. For example, if you want to exclude a trading partner from using FASP. You can also configure the remote node record later.
    Example:
    myRmtNodePartner:\
    … :fasp=yes:\ :fasp.policy=fair:\ :fasp.bandwidth=1GB:\ :fasp.filesize.threshold=1GB:\
    Parameter Value
    fasp Optional. Valid values are yes and no. Enables FASP.
    • If set to no, files sent to this remote node will not use FASP.
    • If set to yes, files sent to this remote node will default to use FASP instead of TCP/IP.
    • This setting can be overridden by the process parameters.
    • The remote server must have FASP enabled.
    fasp.filesize.threshhold Optional. Used to restrict small files from being sent using FASP.
    • If the file is greater than or equal to the stated value, the Connect:Direct server sends the file using FASP. Otherwise, it is sent using TCP/IP.
    • Default is 1GB.
    • You can use KB, MB, or GB designators. If no designator is included, the system uses bits.
    • This setting can be overridden by the process parameters.
    fasp.bandwidth Optional. Default is as stipulated in the FASP license key. Specifies how much bandwidth each transfer can use.
    • Default value can be changed, but cannot exceed the bandwidth specified in the license key.
    • You can use KB, MB, or GB designators. If no designator is included, the system uses bits per second.
    • This setting can be overridden by the process parameters, but cannot exceed the bandwidth specified in the license key.
    fasp.policy Optional. Specifies the fairness of each transfer. Default is fair.
    • This setting can be overridden by the process parameters.
    • Valid values are:
      • Fixed - FASP attempts to transfer at the specified target rate, regardless of the actual network capacity. This policy transfers at a constant rate and finishes in a guaranteed amount of time. This policy typically occupies a majority of the network's bandwidth, and is not recommended in most file transfer scenarios.
      • Fair - FASP monitors the network and adjusts the transfer rate to fully utilize the available bandwidth up to the maximum rate. When other types of traffic build up and congestion occurs, FASP shares bandwidth with other traffic fairly by transferring at an even rate. This is the best option for most file transfer scenarios.
      • High - FASP monitors the network and adjusts the transfer rate to fully utilize the available bandwidth up to the maximum rate. When congestion occurs, a FASP session with high policy transfers at a rate twice of a session with fair policy.
      • Low - Similar to Fair mode, the Low (or Trickle) policy uses the available bandwidth up to the maximum rate as set in the Aspera license file. When congestion occurs, the transfer rate is decreased all the way down to the minimum rate as set in the Aspera license file.

FASP Messages

Use the following table to obtain FASP error message information.

Note: Long text message files for these message IDs can be viewed using the Connect:Direct Requester Message Lookup utility.
Non-Detailed Statistics Mode (Message ID only) Detailed Statistics Mode
FASP001E FASP001E: FASP server session creation failed.
FASP002E FASP002E: FASP client session creation failed.
FASP003E FASP003E: FASP could not be initialized.
FASP004E FASP004E: Lock timeout.
FASP005E FASP005E: Memory allocation failure.
FASP006E FASP006E: Condition wait timed out.
FASP007E FASP007E: No FASP listen ports available.
FASP008E FASP008E: FASP disabled due to file size &FILESIZE < threshold &THRESHOLD
FASP009E FASP009E: FASP session terminated unexpectedly.
FASP010E FASP010E: SNODE refused FASP, FASP disabled.
FASP011E FASP011E: FASP CRC verification failed.
FASP020E FASP020E: Session Manager received invalid FASP control message.
FASP021E FASP021E: FASP control message fragmented or invalid.
FASP022E FASP022E: Session Manager failed to receive FASP control message.
FASP023E FASP023E: The FASP control message to send exceeds the buffer size.
FASP024E FASP024E: Session Manager failed to send FASP control message.
FASP030E FASP030E: FASP license file not found.
FASP031E FASP031E: FASP license file expired.
FASP032E FASP032E: FASP license in error.
FASP033E FASP033E: FASP license is malformed.
FASP034E FASP034E: FASP license is malformed.
FASP035E FASP035E: FASP License file at &LOCATION will expire in &VALUE day(s).
FASP040E FASP040E: FASP initialization failed - remote &TYPE &NODE. Error=&ERROR.
FASP041E FASP041E: FASP initialization failed - local &TYPE &NODE. Error=&ERROR.
FASP042E FASP042E: FASP initialization failed.

Monitoring FASP transactions

You can view the Copy Termination Record (CTRC) for detailed statistics. For example, you can verify FASP was used, what bandwidth was used, and which policy was used.

In the example below, note the following explanations:
  • FASP=>Y indicates that FASP was used to transfer this file. FASP=>N would indicate TCP/IP was used.
  • FSPL=>FAIR is the policy negotiated for this file transfer.
  • FSBW=>1000000000 is the bandwidth negotiated for this file transfer (in bits per second).
  • FMBC =>2 is the high water mark for the number of FASP buffers used
  • FBCS =>16777216 is the FASP buffer size
  • FSTH =>1073741824 is filesize threshold
  • FSLP =>23708 is listen port used for FASP
Example:
PROCESS RECORD Record Id => CTRC
Completion Code => 0
Message Id => SCPA000I
Short Text => Copy step successful.
Ckpt=>Y Lkfl=>N Rstr=>N Xlat=>N Scmp=>N Ecmp=>N CRC=>N
FASP=>Y  FSPL=>FAIR  FSBW=>10000000000  FMBC=>2  FBCS=>16777216 FSTH=>1073741824
FSLP=>23708

Limitations

The following features cannot be used with FASP and Connect:Direct for UNIX:

  • Firewall navigation source ports should not be used with FASP
  • Silent installation does not support the FASP configuration parameters