Heterogeneous IPSec solution between AIX and Windows

Anto A. John and Akshay Kaushik
Published on August 24, 2010

IPSec (Internet Protocol Security) is a protocol for securing IP communication. It authenticates and encrypts each IP packet flowing through the network. This is particularly important when you try to interoperate between disparate systems without the worry of security risks between them.

A virtual private network (VPN) is an extension of an enterprise's private intranet across a public network such as the Internet, creating a secure private connection essentially through a private tunnel. VPNs securely convey information across the Internet connecting remote users, branch offices, and Business Partners into an extended corporate network.

In a VPN, there are security exposures everywhere along an end-to-end path: on the dial-up link, in an ISP's access box, in the Internet, in the firewall or router, and even in the corporate intranet. Hence, there arises a need for this VPN to be protected. The Internet Engineering Task Force has recommended that the tunnel traffic should be protected with the IPSec protocols.

Heterogeneity on end points in a VPN is extremely high, and it demands that the IPSec solution should work well with heterogeneous systems and environments. Hence, this article deals with the AIX IPSec solution and their configuration with Windows as another end point to showcase the heterogeneous capability of this solution.

Configuring Windows 2000 for IPSec

The configuration of IPSec for Windows 2000 requires the creation of the tunnel parameters and the kind of encryption using the IPSec snap-ins.

Create a custom MMC console

The Windows 2000 machine can be configured and monitored using the MMC (Microsoft Management Console). IPSec snap-ins need to be added to this console.

From the Windows desktop, click Start, click Run, and in the Open textbox type mmc. Click OK.
On the File menu, click Add/Remove Snap-in.
In the Add/Remove Snap-in dialog box, click Add.
In the Add Standalone Snap-in dialog box, click IP Security Policy Management, and then click Add.
Verify that Local Computer is selected, and click Finish.
In the Add Standalone Snap-in dialog box, click IP Security Monitor, and then click Add.
To close the Add Standalone Snap-in dialog box, click Close.
To close the Add/Remove Snap-in dialog box, click OK.
Save this as IPSec.msc for future use.

IPSec Snap-in

View image at full size

Creating IPSec policies

In this step, we create and define the IPSec policies using the Windows machine that negotiates with the other machines.

In the MMC Console, right-click IP Security Policies on Local Machine, and then click Create IP Security Policy. The IP Security Policy Wizard appears.
IP Security Policy Wizard
View image at full size
Click Next.
Type Policy1 as the name of your policy, and click Next.
Clear the Activate the default response rule checkbox, if you would like to set your own rules, and then click Next.
Make sure the Edit Properties checkbox is selected (it is by default), and then click Finish.
IPSec Policy1 created
View image at full size
In the Properties dialog box for the policy you have just created, ensure that Use Add Wizard checkbox in the lower-right corner is selected, and then click Add to start the Security Rule Wizard.
Click Next to proceed through the Security Rule Wizard, which you started at the end of the previous section.
Select This rule does not specify a tunnel, (selected by default) and then click Next.
Select the radio button for All network connections, (selected by default) and click Next.

Creating filter rules

In the IP Filter List dialog box, click Add. An empty list of IP filters is displayed. Name your filter Policy1 Filter List.
IP Filter List
View image at full size
Policy1 Filter List
View image at full size
Make sure Use Add Wizard is selected in the center-right area of the screen and then click Add. This starts the IP Filter Wizard.
Click Next to continue.
Accept My IP Address as the default source address by clicking Next.
Choose A Specific IP address from the drop-down list box; enter your Partners IP Address. Here, you can make IPSec communicate with multiple hosts, as well by defining a subnet, and then click Next.
Click Next to accept the protocol type of Any.
Make sure the Edit Properties checkbox is cleared (this is the default setting), and click Finish.
Click Close to leave the IP Filter List dialog box and return to the New Rule Wizard.
In the Filter List dialog box, select the radio button next to Policy1 Filter List.
Policy1 Filter List created
View image at full size
Click Next for configuring filter action.

Configuring filter action

In this section, we define the different actions which the filters perform.

In the Filter dialog shown in Filter Action figure, click to select the Use Add Wizard checkbox, and then click Add.
Filter Action
View image at full size
Click Next to proceed through the Filter Action Wizard.
Name this filter action Policy1 Filter Action and click Next.
In the Filter Action General Options dialog box, select Negotiate Security, and then click Next.
Click Do not communicate with computers that do not support IPSec from the next wizard page, and then click Next to secure your machine from intruders.
Select Custom from the list of security methods, and then click settings. This section gives you opportunity to select whether you would like to have a security method with AH (Authentication Header) or with ESP (Encapsulating Security Payload).
Select Encryption algorithm and hashing algorithm you want to use in your IPSec tunnels to encrypt the data. Click OK to come out of Custom Settings.
Click Next.
Selecting security methods
View image at full size
Make sure the Edit Properties checkbox is cleared (this is the default setting), and then click Finish to close this wizard.
In the Filter Action dialog, click the radio button next to Policy1 Filter Action, and then click Next.
In the Authentication method, select the radio button next to Use this string to protect the key exchange (preshared key). You can also specify the certificates if you don't wish to use the symmetric preshared keys.
Give the preshared key you want to use for authentication in IPSec tunnel (for example 12345) and click Next.
Make sure the Edit properties checkbox is cleared (this is the default setting) and then click Finish. You have just configured the filter action that will be used during negotiations with your partner. Note that you can re-use this filter action in other policies.
In the Properties page that is now displayed, click Close. You have successfully configured an IPSec Policy.
IPSec Policy1 created
View image at full size

Assigning policy

Right click on the policy you have just created and click Assign.

Policy1 assigned as IPSec Security Policy

View image at full size

Configuring AIX for IPSec

For the IPSec negotiation to go through, we need to open up a few ports and protocols on the firewall. They are:

Ports and protocols for IPSec

   - UDP port 500 (for ISAKMP traffic)   
   Protocol  
   - IP Protocol 50 (for ESP traffic)                        
   - IP Protocol 51 (for AH traffic)
   - And any other port according to your environment.

AIX IPSec prerequisites

Install AIX IPSec software and put on latest IPSec patches:

IPSec file sets

      bos.msg.en_US.net.ipsec                                            
      bos.net.ipsec.keymgt                                               
      bos.net.ipsec.rte                                                  
      bos.net.ipsec.websm          
      bos.crypto-priv
      gskak.rte

To start the IP security on AIX, run the following command:

Smitty ipsec4 ------->start/stop IP Security-------->start IP Security

Start IP security

Type or select values in entry fields.
Press Enter AFTER making all desired changes.
                                           [Entry Fields]
Start IP Security                    [Now and After Reboot]
Deny All Non_Secure IP Packets                 [no]

Press Enter to start the IP security. Run the following command to check the state of the IPSec devices.

#lsdev -Cc ipsec

Both the devices should be in the available state (ipsec_v4 and ipsec_v6).

 # lsdev -Cc ipsec
ipsec_v4 Available  IP Version 4 Security Extension
ipsec_v6 Available  IP Version 6 Security Extension

To configure the IPSec on AIX, we first need to create the IPSec configuration file. This file should be in XML file format.

Sample XML file (Save the file with the name IPSECpolicy1)

<?xml version="1.0"?>
<AIX_VPN
      Version="2.0">
   <IKEProtection
         IKE_Role="Both"
         IKE_XCHGMode="Main"
         IKE_KeyOverlap="10"
         IKE_Flags_UseCRL="No"
         IKE_ProtectionName="P1Pol"
         IKE_ResponderKeyRefreshMaxKB="200"
         IKE_ResponderKeyRefreshMinKB="1"
         IKE_ResponderKeyRefreshMaxMinutes="480"
         IKE_ResponderKeyRefreshMinMinutes="1">
      <IKETransform
            IKE_Hash="MD5"
            IKE_DHGroup="1"
            IKE_Encryption="DES-CBC"
            IKE_KeyRefreshMinutes="480"
            IKE_AuthenticationMethod="Preshared_key"/>
   </IKEProtection>
   <IKETunnel
         IKE_TunnelName="P1"
         IKE_ProtectionRef="P1Pol"
         IKE_Flags_AutoStart="Yes"
         IKE_Flags_MakeRuleWithOptionalIP="No">
      <IKELocalIdentity>
         <IPV4_Address
               Value="Local AIX Host IP"/>
      </IKELocalIdentity>
      <IKERemoteIdentity>
         <IPV4_Address
               Value="Remote Windows Server IP"/>
      </IKERemoteIdentity>
   </IKETunnel>
   <IKEPresharedKey
         Value="12345"
         Format="ASCII">
      <IKEPresharedRemoteID>
	  <PK_IPV4_Address
               Value="Remote Windows Server IP"/>
      </IKEPresharedRemoteID>
   </IKEPresharedKey>
   <IPSecProposal
         IPSec_ProposalName="P2Prop">
      <IPSecAHProtocol
            AH_KeyRefreshKB="0"
            AH_Authentication="AH_MD5"
            AH_EncapsulationMode="Transport"
            AH_KeyRefreshMinutes="580"/>
      <IPSecESPProtocol
            ESP_Encryption="ESP_DES"
            ESP_KeyRefreshKB="0"
            ESP_Authentication="HMAC-MD5"
            ESP_EncapsulationMode="Transport"
            ESP_KeyRefreshMinutes="580"/>
   </IPSecProposal>
   <IPSecProtection
         IPSec_Role="Both"
         IPSec_KeyOverlap="10"
         IPSec_ProposalRefs="P2Prop "
         IPSec_ProtectionName="P2Pol"
         IPSec_InitiatorDHGroup="0"
         IPSec_ResponderDHGroup="NO_PFS GROUP_1 GROUP_2 GROUP_5"
         IPSec_Flags_UseLifeSize="No"
         IPSec_Flags_UseCommitBit="No"
         IPSec_ResponderKeyRefreshMaxKB="200"
         IPSec_ResponderKeyRefreshMinKB="1"
         IPSec_ResponderKeyRefreshMaxMinutes="220"
         IPSec_ResponderKeyRefreshMinMinutes="1"/>
   <IPSecTunnel
         IKE_TunnelName="P1"
         IPSec_TunnelName="P2"
         IPSec_ProtectionRef="P2Pol"
         IPSec_Flags_OnDemand="Yes"
         IPSec_Flags_AutoStart="Yes">
      <IPSecLocalIdentity>
         <IPV4_Address
               Value="Local AIX Server IP"/>
      </IPSecLocalIdentity>
      <IPSecRemoteIdentity>
         <IPV4_Address
               Value="Remote Windows Server IP"/>
      </IPSecRemoteIdentity>
   </IPSecTunnel>
</AIX_VPN>

Update new IPSec configuration in the IKE database

We first need to remove the previous IPSec configuration in the IKE database, and then put the new configuration file in the ikedb.

To remove the previous configuration, run the following command:

# ikedb -x
P1_ITD database created successfully
P2_ITD database created successfully
P1_PREKEY database created successfully
PROPOSAL_LIST database created successfully
PROPOSAL database created successfully
POLICY database created successfully
GROUP database created successfully
NDBM:/etc/ipsec/inet/DB/privkey

To put the new configuration file in the database, run the following command:
```
# ikedb -p IPSECpolicy1
```

Check if all the three daemons (tmd, isakmpd and cpsd) are running. The tmd daemon takes care of the tunnel management, and the isakmpd daemon takes care of the IKE negotiation. If we are not using certificates for authentication, there is no need for the cpsd daemon to run.

To start the daemons, run the following command:

#  startsrc -g ike
0513-059 The cpsd Subsystem has been started. Subsystem PID is 434304.
0513-059 The tmd Subsystem has been started. Subsystem PID is 315554.
0513-059 The isakmpd Subsystem has been started. Subsystem PID is 401504.

Run the following command to check if the daemons are started or not. If the daemon is started, the status of that daemon should be active.

# lssrc -g ike
Subsystem         Group            PID          Status
cpsd              ike              241894           active
tmd               ike              315550           active
isakmpd           ike              319648           active

Run the following command to check if any IPSec tunnel is active:

# ike cmd=list
No tunnels match your request.

If you do not find the tunnels between the machines you actually intend to have the tunnel, then run the following command to activate the tunnels:

# ike cmd=activate
Phase 2 tunnel 1 activate request initiated.

Now the ike cmd command should list the state of the tunnels for you.

# ike cmd=list
Phase  Tun Id  Status      Local Id                        Remote Id
1      1       Dormant         9.124.101.138                9.124.101.175
2      1       Dormant         9.124.101.138                9.124.101.175

We need to ping the remote host to activate the tunnels. One or two ping request may be denied until the tunnels become active. The requests will be successful from then on.

# ping 9.124.101.175
PING 9.124.101.175 (9.124.101.175): 56 data bytes
ping: sendto: Permission denied
ping: wrote 9.124.101.175 64 chars, ret=-1
64 bytes from 9.124.101.175: icmp_seq=1 ttl=255 time=0 ms
64 bytes from 9.124.101.175: icmp_seq=2 ttl=255 time=0 ms
64 bytes from 9.124.101.175: icmp_seq=3 ttl=255 time=0 ms

Now you have created a successful AIX to Windows IPSec tunnel that can be further used for secure communication over the network.

Conclusion

This article showcases the ability of AIX IPSec to work across heterogeneous environments. Similar to the Windows IPSec configuration reviewed in this article, you can try using other operating systems to communicate securely with AIX using IPSec. Doing so can provide greater security in an insecure public network with heterogeneous systems.

Downloadable resources

PDF of this content

Contents