Secure file transfer in a heterogeneous environment

Using the AIX V6.1 security feature

File transfer is an essential and important activity in the day-to-day computing world. Security lapses during file transfer can invite leak important data to the external world. As a result, securing FTP is of primary importance. Hence, in AIX® V6.1, IBM® has introduced a secure flavor of FTP (and ftpd) based on OpenSSL, using Transport Layer Security (TLS) to encrypt both the command and the data channels of file transfer. This article shows the advantage of using this AIX V6.1 feature and its usage between AIX and other heterogeneous systems that already support this feature. This article focuses on AIX secure FTP with a Windows® server.


Anto A. John (, System Software Engineer, IBM

author photoAnto John is an AIX development support specialist working for IBM India Software Labs in Bangalore, India. He works on AIX security components (security library), load modules (LDAP and Kerberos), and new AIX 6.1 security features (EFS). For the past three years, he has worked on AIX security components as well as open source components like OpenSSH and OpenSSL. He graduated from BITS Pilani with a Masters of Engineering degree in computer science. You can contact him at

14 April 2009

Also available in Chinese


Transport Layer Security (TLS) is a cryptographic protocol that provides secure communication between clients and servers. When a server and client communicate, TLS ensures that no third party may eavesdrop or tamper with any message. TLS is the successor to the Secure Sockets Layer (SSL). This enables any user on the system to exchange files in a secure manner, as long as their counterpart offers this extension as well.

You might wonder if you already have a secure file transfer protocol, or FTP, (sftp) which comes with openSSH, why would you opt for a new way of securing the FTP? While it may seem that using secure FTP only and no secure telnet might not be most desirable, this method is in fact a sensible alternative for environments where you are not able to use OpenSSH. For example, if your most trusted systems run on a dedicated and segregated network, it makes perfect sense to use telnet for remote access within that very network zone (or working from the console). But even in such scenarios, you might very well have the need to transfer data from or to this secure zone, which can be accomplished now using secure FTP.

Another scenario could be that you are using OpenSSH already but you still have to exchange data with outside systems that do not support any form of SSH (thus neither scp nor sftp). Most often, such (legacy) systems offer "FTP via SSL" (often called ftps) instead.

Requirements for secure FTP

TLS relies on Secure Socket Layer, which needs OpenSSL to be installed on the AIX system (ftps depends on the libssl.a and libcrypto.a libraries). The latest version of IBM-supported OpenSSL can be obtained from

The secure communication between the FTP client and server makes use of the certificates, a combination of a public-private key pair. Hence, the creation of certificates in-house or the import of certificates from commercial certifying authorities is the second requirement for setting up FTP over SSL.

Setting up FTP between AIX and Windows

There are four steps to set up FTP between AIX and Windows:

The article looks into each of these activities separately.

Creation of certificates

As stated previously, certificates need to be created for negotiation between the FTP server and the client if there is not already a commercial certificate.

First, install OpenSSL on the Windows machine if certificates are created at this machine. If not, the commercial certificates just need to be brought into this machine.

Create a root-level private key and root-level certificate request (holding the public key).

Listing 1. Root-level certificate request
c:\>cd c:\openssl\bin\
openssl req -newkey rsa:2048 -sha1 -keyout root_key.pem -out root_req.pem
Generating a 2048 bit RSA private key
writing new private key to 'root_key.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:TX
Locality Name (eg, city) []:Austin
Organization Name (eg, company) [Internet Widgits Pty Ltd]:IBM
Organizational Unit Name (eg, section) []:AIX
Common Name (eg, YOUR name) []:windows
Email Address []

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:1234
An optional company name []:ibm

Generate the certificate for root (valid for approximately 10 years) by self-signing it:

Listing 2. Root certificate creation
openssl x509 -req -days 3650 -in root_req.pem -signkey root_key.pem -out root_cert.pem
Signature ok
Getting Private key
Enter pass phrase for root_key.pem:

You can have a look at your root certificate just to make sure everything's right by using:

Listing 3. Root certificate
openssl x509 -in root_cert.pem -text -noout
        Version: 1 (0x0)
        Serial Number:
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=TX, L=Austin, O=IBM, OU=AIX, 
            Not Before: Nov 12 19:54:40 2008 GMT
            Not After : Nov 10 19:54:40 2018 GMT
        Subject: C=US, ST=TX, L=Austin, O=IBM, OU=AIX, 
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha1WithRSAEncryption

Now you will create an RSA key for the Windows FTP server without a PEM passphrase; hence, you will use a different command than the one used previously. It is important not to use any passphrases on such server keys. Otherwise, it would be required to input that passphrase every time the key gets used (which is impossible to accomplish when ftpd is using it).

Listing 4. Key for Windows FTP server
openssl genrsa 2048 
Loading 'screen' into random state - done
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x10001)

Copy the above key content into a file named server_key.pem.

Next, create a certificate request for the key you just created (including its public key):

Listing 5. Certificate request for the created key
openssl req -new -key server_key.pem -out server_req.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:TX
Locality Name (eg, city) []:Austin
Organization Name (eg, company) [Internet Widgits Pty Ltd]:IBM
Organizational Unit Name (eg, section) []:AIX
Common Name (eg, YOUR name) []:server
Email Address []

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:1234
An optional company name []:ibm

Next, sign the server key request with the root CA's private and self-signed public key. This creates the server certificate (again, valid approximately for 10 years):

Listing 6. Step 6: Server certificate creation
openssl x509 -req -days 3650 -in server_req.pem -CA root_cert.pem -CAkey root_key.pem 
-CAcreateserial –out server_cert.pem
Signature ok
Getting CA Private Key
Enter pass phrase for root_key.pem:

In order to make server configurations easier as well as the distribution of certified key files, it is handy to have the server key, the server certificate, and the root certificate in one single file (OpenSSL supports this). So combine all three files into one file. This file should be protected with respective file permissions.

Listing 7. Single server file
Copy the contents of  server_key.pem server_cert.pem root_cert.pem 
into the file server.pem

Export the server certificate into the .pfx format.

Listing 8. Convert the server certificate to pfx format
openssl pkcs12 -export -out server.pfx -in server.pem -name "server cert pfx"
Enter Export Password:
Verifying - Enter Export Password:

Now you have the server certificate in the .pfx format, which Windows accepts. It is very important that you add this certificate to the Microsoft® Management console's trusted certificate repository. Failure to do so will result in making the certificate untrusted during the usage.

The Microsoft Management Console (MMC) can be accessed by Start -> Run -& type mmc .

The MMC has an option to add the certificates as snap-ins.

Figure 1. Microsoft Management Console
Screen shot of Microsoft Management Console
Figure 2. Adding snap-ins
Screen shot of adding snap-ins

Once the Snap-in "certificate" is added to the console, you get the certificate categories as listed below. Select the trusted root certificate or any other certificate category accordingly and import the server certificate that you had created into it.

Figure 3. Importing the certificate
Screen shot showing how to import the certificate

The import of the certificate will open up the Certificate Import Wizard, following which the import of certificate becomes successful.

Figure 4. Certificate Import Wizard
Screen shot of Certificate Import Wizard

You have now changed the OpenSSL certificate for the FTP purpose into a trusted certificate in the Windows machine.

Figure 5. Certificate added successfully to the snap-in
Screen shot showing how a certificate is added successfully to the snap-in

Now open up the Internet Information Services (IIS) manager and add this trusted certificate as a part of the the ISS server certificate database and now it can be used for the different services of IIS, which includes secure FTP.

Figure 6. Server Ceriticate repository
Screen shot of Server Ceriticate repository
Figure 7. Import into Server Certificate repository
Screen shot of an import into Server Certificate repository

Importing the server certificate into this repository makes it usable for the secure FTP service. Also, note that the certificate is OK and is trusted. After the certificate is successfully added, you have to 'apply' the changes in the IIS Manager for the changes to take effect.

Figure 8. Imported certificate
Screen shot of imported certificate

Setting up FTP services on a Windows server

To set up FTP on the Windows side, you have to open the console for Internet Information server (IIS). IIS can be accessed through Start -> Administrative Tools -> Internet Information Services(IIS) Manager.

Figure 9. Internet Information services (IIS) Manager
Screen shot of Windows Start menu, selecting Internet Information services (IIS)

Create a new site for this purpose and the work area will create the FTP-specific services. In this case, create a site called ssl_ftp_site.

Figure 10. Adding an FTP site
Screen shot of adding a ftp site

You can assign specific IP address and ports for which this FTP server needs to be associated. In this step, there is an option that asks you to specify the certificate. If the certificate is created and ready for usage, it can be specified now or could be left for later configuration.

Figure 11. Binding and SSL settings
Screen shot of binding and SSL settings

This step configures the FTP servers authentication mode and the users who are allowed, along with their read/write permissions and completes the configuration of the FTP site in the IIS.

Figure 12. Authentication and authorization information
Screen shot of authentication and authorization information

The panel at the right side under the "Manage FTP Site" section helps you to either stop or run the service. It is always important to stop the server and restart it for any changes to take effect.

Figure 13. Site ready for use
Screen shot of site ready for use

Now you must set the FTP SSL settings for the site, which is available on the workspace of the FTP site home.

Figure 14. FTP SSL settings
Screen shot of FTP SSL settings

Again, do not forget to stop the server and restart it for the changes to take effect.

Configuration of FTP at the AIX client

There is one further step of security that can be established with such self-signed certificates: The rootCA's certificate needs to be copied to every client and referred to in their ~/.ftpcnf files of any user wishing to use encrypted and authenticated FTP sessions.

Adjust the user's .ftpcnf file to point to the CA certificate by only changing this one line:

CA_PATH ./root_cert.pem

Data transfer using secure FTP

You can use /usr/bin/ftp to connect to the FTP daemon on any remote system, whether this target host is TLS-enabled or not. Once you have completed the TLS setup on your server, in order to transfer files to or from that server, you can issue the following command.

# ftp -s –M remote_host

This will establish an encrypted session without any modifications needed on the client side. If the certificate, however, is self-signed, the client has to blindly trust the server, so the FTP command will display the most important data of the certificate it received during the TLS handshake to leave the decision up to the user whether he wants to connect or not.

The command ftp –s remote_host will work only for AIX-to-AIX machines. One has to use the ftp –s –M remote_host option to get the secure FTP working between AIX and non-AIX machines.


This article covered the various aspects of the secure FTP setup, which includes the FTP service configuration on the Windows server side and the FTP configuration on the client side. This feature adds a very important security capability to the AIX operating system and helps it to harness the similar functionality available with other operating systems, tthus helping the customer to have a secure file transfer in a heterogenous environment.



Get products and technologies



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks

Zone=AIX and UNIX
ArticleTitle=Secure file transfer in a heterogeneous environment