IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 1

Generating certificates in AIX and importing certificates to Windows for IKE IPsec tunnels

Using AIX 6.1 or later and Microsoft Windows 2012

Content series:

This content is part # of # in the series: IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 1

Stay tuned for additional content in this series.

This content is part of the series:IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 1

Stay tuned for additional content in this series.

Internet Protocol Security (IPsec) is a widely used method to secure network transmission at the Internet Protocol (IP) layer. The IP layer is a strategic place in the network stack to secure communication as all traffic from lower and higher layers pass through it. Also, security provided at this layer is independent of the lower layers and is transparent to the higher layers. In other secure transmission protocols such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), the applications need to be enabled to make use of the security protocols. But with IPsec, the applications are not aware of the underlying encryption or decryption at the IP layer.

IPsec is supported between heterogeneous systems as well. Data centers usually have a heterogeneous set of systems, that is, systems belonging to different vendors. For example, a data center could have an IBM® AIX® Enterprise server hosting an application server with Microsoft® Windows® desktop systems as the endpoint clients to this AIX server. We can still use IPsec to secure communications in such heterogeneous environments. The only requisite being that the IPsec implementations on these different systems need to be IPsec, Internet Key Exchange (IKE), and Request for Comments (RFC) compliant.

This tutorial discusses certificate generation on AIX using the IBM Global Security Kit (GSKit) and importing the certificates to Windows operating system. This is the first step in establishing the IKE tunnels between AIX (6.1 / 7.1 / 7.2) and Microsoft Windows 2012 using certificates. These steps are required when establishing IKEv1 or IKEv2 tunnels with certificates. Table 1 provides a short description of the main topics covered in this tutorial and what they entail.

Table 1. Main topics covered
Certificate generation This section describes how to generate the certificates on AIX.
Importing certificates to Windows This section describes how to import the certificates to the Windows 2012 system.

Certificate generation

This section describes how to generate certificates on AIX and in the later section we will see how to import these in Windows 2012. The certificate generation steps are common to both IKEv1 and IKEv2. To generate certificates on AIX, we use the IBM GSKit V8 file sets.

Figure 1. lslpp output querying GSKit8 file sets

The absolute path for gsk8capicmd on AIX is /usr/opt/ibm/gsk8/bin/gsk8capicmd. The installation usually creates the links from /usr/bin/gsk8capicmd to /usr/opt/ibm/gsk8/bin/gsk8capicmd.

The following is the set of key databases (kdbs/ key dbs) and certificates that we will be creating as part of the configuration.

Figure 2. Relation between key databases and the certificates in them

AIX IPsec refers to a key database (DB), named ikekey.db, to check for certificates to be used as part of IKE negotiations. So first, let's create this key DB on the AIX system. For this documentation, all the certificates and databases will be first created under a directory named /GSK_CERTS. Later, they will be moved under the right directory.

gsk8capicmd -keydb -create -db /GSK_CERTS/ikekey.kdb -pw 123456 -type cms -stash

In the above command, we are using 123456 as the password for keydb. It would be better to have a stronger password other than the first six consecutive positive integers. But in this tutorial, let us just use this password for the sake of simplicity.

This creates four files: ikekey.crl, ikekey.kdb, ikekey.rdb and ikekey.sth.

Next, we need to create the root certificate authority (root CA) in a separate database. The following command creates key DBs for the root CA. It creates rootCA.crl, rootCA.kdb, rootCA.rdb, and rootCA.sth. under /GSK_CERTS.

gsk8capicmd -keydb -create -db   /GSK_CERTS/rootCA.kdb -pw 123456 -type cms -stash

The following command creates the root CA in the new rootCA.kdb key DB.

gsk8capicmd -cert -create -db /GSK_CERTS/rootCA.kdb -pw 123456 -size 1024 -dn "C=IN,O=IBM,CN=ipsecroot" -label Root_CA_Full -default_cert  yes -ca true

You will now see the following files under the /GSK_CERTS directory.

Figure 3. Files under /GSK_CERTS directory

When we add the root CA to an AIX system's ikekey.db key database, we must extract and add the public part of the root CA. If both private and public keys are present in Root_CA certificate, then AIX IPsec will treat it as a normal certificate and not as the root CA.

To extract the public part of the root CA, run the following command:

gsk8capicmd -cert -extract -db /GSK_CERTS/rootCA.kdb -stashed -label Root_CA_Full -format ascii -target rootCA_Pub.arm

Add the above extracted rootCA_Pub.arm to ikekey.db using the following command:

gsk8capicmd -cert -add -db /GSK_CERTS/ikekey.kdb -stashed -label Root_CA -file rootCA_Pub.arm -format ascii -trust enable

If we are to query the ikekey.db key databases now, we will see the newly added Root_CA there:

gsk8capicmd -cert -list -db /GSK_CERTS/ikekey.kdb -pw 123456
Figure 4. Listing the certificates in ikekey.kdb

The following command creates a certificate for the AIX host:

gsk8capicmd -certreq -create -db /GSK_CERTS/ikekey.kdb -pw 123456 -label Test_Cert2 -dn "C=IN,ST=KA,L=BA,O=IBM,OU=ISL,CN=test2" -size 1024 -file /GSK_CERTS/cert2_csr.arm

Next let's sign the Test_Cert2 certificate with Root_CA. But remember that we have only the public key part of root CA (Root_CA) in ikekey.kdb and the full root CA in rootCA.kdb. So, while signing the certificate we created, we need to use Root_CA_Full.

gsk8capicmd -cert -sign -db /GSK_CERTS/rootCA.kdb -pw 123456 -label Root_CA_Full -target /GSK_CERTS/Test_Cert2.cer -format ascii -expire 100 -file /GSK_CERTS/cert2_csr.arm
Figure 5. ikekey files and their sizes under /GSK_CERTS directory

After signing the certificate, the certificate is now present in the ikekey.rdb otherwise known as the ikekey request database (rdb). It still needs to be received into the ikekey.kdb key database. The following command receives the key into the ikekey.kdb key database.

gsk8capicmd -cert -receive -file /GSK_CERTS/Test_Cert2.cer -db /GSK_CERTS/ikekey.kdb  -pw 123456 -type cms -format ascii
Figure 6. Second listing of ikekey files and their sizes

Until the certificate is received, this newly created certificate will be residing in ikekey.rdb. After the certificate has been received, we can see an increase in the size of ikekey.kdb as shown in Figure 5 and Figure 6.

Figure 7. Second listing of certificates in ikekey.kdb

We now have Root_CA and Test_Cert2 in ikekey.kdb. The ikekey.* databases will be residing in the AIX host. So, let's move all the relevant ikekey.* databases to /etc/security.

/usr/bin/mv  /GSK_CERTS/ikekey* /etc/security

Next, let us create a certificate for the Windows 2012 system. We will be creating this certificate on the AIX system using Root_CA_Full to sign this certificate as we did earlier. But as this certificate would belong to Windows 2012, let us create a new DB to keep it simple. We will call this Rikekey.kdb where "R" refers to remote system.

gsk8capicmd -keydb -create -db   /GSK_CERTS/Rikekey.kdb -pw 123456 -type cms -stash

Next, add Root_CA (with only the public part) to Rikekey.kdb.

gsk8capicmd -cert -add -db /GSK_CERTS/Rikekey.kdb -stashed -label Root_CA -file rootCA_Pub.arm -format ascii -trust enable

Create the remote certificate.

gsk8capicmd -certreq -create -db  /GSK_CERTS/Rikekey.kdb -pw 123456 -label Test_Cert1 -dn "C=IN,ST=KA,L=BA,O=IBM,OU=ISL,CN=test1" -size 1024 -file /GSK_CERTS/cert1_csr.arm

gsk8capicmd -cert -sign -db /GSK_CERTS/rootCA.kdb -pw 123456 -label Root_CA_Full -target /GSK_CERTS/Test_Cert1.cer -format ascii -expire 100 -file /GSK_CERTS/cert1_csr.arm

gsk8capicmd -cert -receive -file /GSK_CERTS/Test_Cert1.cer -db /GSK_CERTS/Rikekey.kdb  -pw 123456 -type cms -format ascii

Now we have a new certificate (Test_Cert1) in Rikekey.kdb. We need to register this certificate with Windows. But before that, we need to have Test1 and Root_CA in a format that is acceptable to Windows. Let's do that.

gsk8capicmd -cert -export -db /GSK_CERTS/Rikekey.kdb -pw 123456 -label Test_Cert1  -type cms -target Test1.p12 -target_type p12 -target_pw 123456

gsk8capicmd -cert -export -db /GSK_CERTS/Rikekey.kdb -pw 123456 -label Root_CA  -type cms -target Root.p12 -target_type p12 -target_pw 123456

Use FTP to transfer Test1.p12 and Root.p12 to Windows. Take care to transfer using the bin mode.

Importing certificates to Windows 2012

You need to perform the following steps to import certificates to a Windows 2012 system:

  1. On the Windows 2012 system, click Start and then click MMC.
  2. In the MMC console, click File -> Add/Remove Snap-in.
    Figure 8. Adding a new snap-in in MMC
  3. Search for the snap-in named Certificates and click Add. Then, click OK.
    Figure 9. Adding the Certificates snap-in
  4. In the Certificates snap-in dialog box, select Computer account and click Next.
    Figure 10. Select the account that will be managed by Certificates snap-in
  5. In the Select Computer dialog box, select Local Computer and click Finish.
    Figure 11. Select the computer that the snap-in will manage
  6. In the main window, click OK. After adding the certificates plug-in, you should be able to view it as shown in Figure 12.
    Figure 12. Certificates snap as seen on local computer
  7. To import certificates, expand Certificates, right-click Personal, and click All Tasks -> Import.
    Figure 13. Import certificates
  8. On the welcome page of the Certificate Import Wizard, click Next.
    Figure 14. Certificate Import Wizard window
  9. Browse for the Test.p12 certificate and click Next.
    Figure 15. Browsing to get the Test.p12 certificate

    The Test1.p12 certificate is available under C:\Users\Administrator\, and you can find this path in Figure 15. This is not the default path. This is the path where you copied the certificate in Windows.

  10. Provide a password for the private key. For simplicity, we use the same password, 123456, as we used on AIX and click Next.
    Figure 16. Certificate Import Wizard: Private key protection
  11. Specify the options to place all the certificates in the Personal store (as shown in Figure 17).
    Figure 17. Selecting Personal store as the certificate store
  12. Verify the settings specified for the certificate and click Finish.
    Figure 18. Verify the store and the certificate to be imported

After Test1 has been successfully imported, you can find it as shown in Figure 19.

Figure 19. Verify that the imported certificate appears in Certificates snap-in

The next step is to import the Root_CA certificate under Trusted Root Certification Authorities. If you have intermediate CA certificates, perform the following steps but remember to import the intermediate CA certificate under Intermediate Certificate Authorities. For more information, refer Related topics. This registers the root CA in the certificate database on Windows.

  1. Right-click Trusted Root Certification Authorization and then click All Tasks -> Import.
    Figure 20. Expanding options under Trusted Root Certification Authorization
  2. Specify Trusted Root Certification Authorities as the store and click Next.
    Figure 21. Selecting the certificate store
  3. Simlar to importing Test1.p12, browse and import Root.p12. Then, click Finish.
    Figure 22. Certificate Import Wizard completion

    We can now see that the root CA has been imported successfully to Windows (refer Figure 23).

    Figure 23. Root CA certificate


This tutorial explained the steps to create certificates on AIX using GSKit and to import to the Windows 2012 operating system. This is the first step in configuring a tunnel using certificates between AIX and Windows. The next steps for configuring tunnels between AIX and Windows operating systems are mentioned in the following tutorials:

The current tutorial and the above tutorials, together, explain four different ways to establish tunnels between AIX 6.1/ 7.1/ 7.2 and Windows 2012 systems. These four methods are setting up IKEv1 and IKEv2 tunnels using pre-shared keys and certificates between AIX and Windows.

The configuration steps for Windows (explained in this tutorial) were tried in the lab and worked for the test team. These steps are not endorsed by Microsoft nor by IBM. You can consider the steps mentioned in this tutorial as a guidance to get you started. There is always light at the end of the tunnel. So, enjoy configuring tunnels!

Related topics

Downloadable resources

Zone=AIX and UNIX
ArticleTitle=IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 1: Generating certificates in AIX and importing certificates to Windows for IKE IPsec tunnels