FTP (remote file system) connection

To access your data with the FTP protocol, create a connection asset for it.

FTP is a standard communication protocol that is used to transfer files from a server to a client on a computer network.

Create an FTP connection

To create the connection asset, you need these connection details based on your deployment:

Common connectivity

  • Connection mode: The connection method configured on the FTP server:
    • Anonymous
    • Basic authentication (with username and password)
    • SFTP Tectia for z/OS: Transfer data sets that are in Multiple Virtual Storage (MVS) format to or from an IBM z/OS mainframe computer. MVS data sets use a period (.) to separate the qualifiers in the data set names. To write to an MVS data set, select Access MVS Dataset and enter the file transfer advice (FTADV) strings in key-value pairs separated by commas. For information, see the Tectia documentation.
    • SSH (SFTP): File transfer over a secure channel that uses the Secure Shell protocol. Also requires username and password.
    • SSL: File transfer that uses File Transport Protocol (FTP), which supports secure transmission via SSL (sslTLSv2) protocol. Also requires username and password.
  • Hostname or IP address
  • Port number of the FTP server

StreamSets

  • Connection mode: The connection method configured on the FTP server:

    • Anonymous
    • Basic authentication (with username and password)
    • SSH (SFTP): File transfer over a secure channel that uses the Secure Shell protocol. Also requires username and password.
    • SSL: File transfer that uses File Transport Protocol (FTP), which supports secure transmission via SSL (sslTLSv2) protocol. Also requires username and password.

  • Hostname or IP address

  • Port number of the FTP server

  • Path: Enter the relative path to the user home directory where the files will be stored.

  • FTPS Mode: This option is only available if you chose the SSL connection mode. Select the FTP encryption negotiation mode. Choose Implicit if the encryption will be used immediately. Choose Explicit if the FTP server supports FTPES and requires the FTP client to negotiate the encryption mode.

  • FTPS Data Channel Protection Level: This option is only available if you chose the SSL connection mode. Choosing Private means that both the communication and data are both encrypted. Choosing Clear means that only the communication is encrypted.

  • Enable Proxy: Choose this option if you want to connect through a proxy. If enabled, fill the following fields:

    • Proxy type: HTTP or SOCKS
    • Proxy URL: Specify the proxy URL where the SFTP, FTP or FTPS server is located.

Credentials

Choose your authentication methods based on your deployment:

Common connectivity

The authentication methods are determined by the Connection Mode chosen:

  • Basic authentication: Choose the username and password.

  • SFTP Tectia for z/OS: Choose which credentials you want to provide:

    • Username and password
    • Username, password, private key and optional key passphrase. If you use an encrypted private key, you will need a key passphrase.
    • Username, private key and optional key passphrase. If you use an encrypted private key, you will need a key passphrase.
    • Access MVS Dataset: An additional property you can enable. When enabled, provide the File Transfer Advice (FTADV) strings as comma-delimitered key-value pairs.

  • SSH (SFTP): Choose which credentials you want to provide:

    • Username and password
    • Username, password, private key and optional key passphrase. If you use an encrypted private key, you will need a key passphrase.
    • Username, private key and optional key passphrase. If you use an encrypted private key, you will need a key passphrase.

If you use a private key, make sure that the key is an RSA private key that is generated by the ssh-keygen tool. The private key must be in the PEM format.

  • SSL: Choose the username and password.
    • SSL certificate: Enter the SSL certificate of the host to be trusted
    • Hostname in SSL certificate: The hostname in SubjectAlternativeName or Common Name (CN) part of the SSL certificate
    • Validate the SSL certificate: Enable this option to validate that the SSL certificate returned by the host is trusted.

StreamSets

The authentication methods are determined by the Connection Mode chosen:

  • Basic authentication: Choose the username and password.

  • SSH (SFTP): Choose which credentials you want to provide:

    • Username and password
    • Username, private key and optional key passphrase. If you use an encrypted private key, you will need a key passphrase.

  • SSL: Choose the username and password.

    • SSL certificate: Enter the SSL certificate of the host to be trusted
    • Hostname in SSL certificate: The hostname in SubjectAlternativeName or Common Name (CN) part of the SSL certificate
    • Validate the SSL certificate: Enable this option to validate that the SSL certificate returned by the host is trusted.

For Private connectivity, to connect to a database that is not externalized to the internet (for example, behind a firewall), you must set up a secure connection. This selection is available for the SSH connection mode only.

Supported file types

The FTP connection supports these file types:  Avro, CSV, Delimited text, Excel, JSON, ORC, Parquet, SAS, SAV, SHP, and XML.

Learn more