Heterogeneous IPSec solution between AIX and Windows
IPSec (Internet Protocol Security) is a protocol for securing IP communication. It authenticates and encrypts each IP packet flowing through the network. This is particularly important when you try to interoperate between disparate systems without the worry of security risks between them.
A virtual private network (VPN) is an extension of an enterprise's private intranet across a public network such as the Internet, creating a secure private connection essentially through a private tunnel. VPNs securely convey information across the Internet connecting remote users, branch offices, and Business Partners into an extended corporate network.
In a VPN, there are security exposures everywhere along an end-to-end path: on the dial-up link, in an ISP's access box, in the Internet, in the firewall or router, and even in the corporate intranet. Hence, there arises a need for this VPN to be protected. The Internet Engineering Task Force has recommended that the tunnel traffic should be protected with the IPSec protocols.
Heterogeneity on end points in a VPN is extremely high, and it demands that the IPSec solution should work well with heterogeneous systems and environments. Hence, this article deals with the AIX IPSec solution and their configuration with Windows as another end point to showcase the heterogeneous capability of this solution.
Configuring Windows 2000 for IPSec
The configuration of IPSec for Windows 2000 requires the creation of the tunnel parameters and the kind of encryption using the IPSec snap-ins.
Create a custom MMC console
The Windows 2000 machine can be configured and monitored using the MMC (Microsoft Management Console). IPSec snap-ins need to be added to this console.
- From the Windows desktop, click Start, click Run, and in the Open textbox type mmc. Click OK.
- On the File menu, click Add/Remove Snap-in.
- In the Add/Remove Snap-in dialog box, click Add.
- In the Add Standalone Snap-in dialog box, click IP Security Policy Management, and then click Add.
- Verify that Local Computer is selected, and click Finish.
- In the Add Standalone Snap-in dialog box, click IP Security Monitor, and then click Add.
- To close the Add Standalone Snap-in dialog box, click Close.
- To close the Add/Remove Snap-in dialog box, click OK.
- Save this as IPSec.msc for future use.
Creating IPSec policies
In this step, we create and define the IPSec policies using the Windows machine that negotiates with the other machines.
- In the MMC Console, right-click IP Security Policies on Local Machine, and then click Create IP Security Policy. The IP Security Policy Wizard appears.
IP Security Policy Wizard
- Click Next.
- Type Policy1 as the name of your policy, and click Next.
- Clear the Activate the default response rule checkbox, if you would like to set your own rules, and then click Next.
- Make sure the Edit Properties checkbox is selected (it is by default), and then click Finish.
IPSec Policy1 created
- In the Properties dialog box for the policy you have just created, ensure that Use Add Wizard checkbox in the lower-right corner is selected, and then click Add to start the Security Rule Wizard.
- Click Next to proceed through the Security Rule Wizard, which you started at the end of the previous section.
- Select This rule does not specify a tunnel, (selected by default) and then click Next.
- Select the radio button for All network connections, (selected by default) and click Next.
Creating filter rules
- In the IP Filter List dialog box, click Add.
An empty list of IP filters is displayed. Name your filter Policy1 Filter
IP Filter List
Policy1 Filter List
- Make sure Use Add Wizard is selected in the center-right area of the screen and then click Add. This starts the IP Filter Wizard.
- Click Next to continue.
- Accept My IP Address as the default source address by clicking Next.
- Choose A Specific IP address from the drop-down list box; enter your Partners IP Address. Here, you can make IPSec communicate with multiple hosts, as well by defining a subnet, and then click Next.
- Click Next to accept the protocol type of Any.
- Make sure the Edit Properties checkbox is cleared (this is the default setting), and click Finish.
- Click Close to leave the IP Filter List dialog box and return to the New Rule Wizard.
- In the Filter List dialog box, select the radio button next to Policy1 Filter List.
Policy1 Filter List created
- Click Next for configuring filter action.
Configuring filter action
In this section, we define the different actions which the filters perform.
- In the Filter dialog shown in Filter Action figure, click to
select the Use Add Wizard checkbox, and then click Add.
- Click Next to proceed through the Filter Action Wizard.
- Name this filter action Policy1 Filter Action and click Next.
- In the Filter Action General Options dialog box, select Negotiate Security, and then click Next.
- Click Do not communicate with computers that do not support IPSec from the next wizard page, and then click Next to secure your machine from intruders.
- Select Custom from the list of security methods, and then click settings. This section gives you opportunity to select whether you would like to have a security method with AH (Authentication Header) or with ESP (Encapsulating Security Payload).
- Select Encryption algorithm and hashing algorithm you want to use in your IPSec tunnels to encrypt the data. Click OK to come out of Custom Settings.
- Click Next.
Selecting security methods
- Make sure the Edit Properties checkbox is cleared (this is the default setting), and then click Finish to close this wizard.
- In the Filter Action dialog, click the radio button next to Policy1 Filter Action, and then click Next.
- In the Authentication method, select the radio button next to Use this string to protect the key exchange (preshared key). You can also specify the certificates if you don't wish to use the symmetric preshared keys.
- Give the preshared key you want to use for authentication in IPSec tunnel (for example 12345) and click Next.
- Make sure the Edit properties checkbox is cleared (this is the default setting) and then click Finish. You have just configured the filter action that will be used during negotiations with your partner. Note that you can re-use this filter action in other policies.
- In the Properties page that is now displayed, click Close. You have successfully configured an IPSec Policy.
IPSec Policy1 created
Right click on the policy you have just created and click Assign.
Policy1 assigned as IPSec Security Policy
Configuring AIX for IPSec
For the IPSec negotiation to go through, we need to open up a few ports and protocols on the firewall. They are:
Ports and protocols for IPSec
- UDP port 500 (for ISAKMP traffic) Protocol - IP Protocol 50 (for ESP traffic) - IP Protocol 51 (for AH traffic) - And any other port according to your environment.
AIX IPSec prerequisites
Install AIX IPSec software and put on latest IPSec patches:
IPSec file sets
bos.msg.en_US.net.ipsec bos.net.ipsec.keymgt bos.net.ipsec.rte bos.net.ipsec.websm bos.crypto-priv gskak.rte
To start the IP security on AIX, run the following command:
Smitty ipsec4 ------->start/stop IP Security-------->start IP Security
Start IP security
Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields] Start IP Security [Now and After Reboot] Deny All Non_Secure IP Packets [no]
Press Enter to start the IP security. Run the following command to check the state of the IPSec devices.
#lsdev -Cc ipsec
Both the devices should be in the available state (ipsec_v4 and ipsec_v6).
# lsdev -Cc ipsec ipsec_v4 Available IP Version 4 Security Extension ipsec_v6 Available IP Version 6 Security Extension
To configure the IPSec on AIX, we first need to create the IPSec configuration file. This file should be in XML file format.
Sample XML file (Save the file with the name IPSECpolicy1)
<?xml version="1.0"?> <AIX_VPN Version="2.0"> <IKEProtection IKE_Role="Both" IKE_XCHGMode="Main" IKE_KeyOverlap="10" IKE_Flags_UseCRL="No" IKE_ProtectionName="P1Pol" IKE_ResponderKeyRefreshMaxKB="200" IKE_ResponderKeyRefreshMinKB="1" IKE_ResponderKeyRefreshMaxMinutes="480" IKE_ResponderKeyRefreshMinMinutes="1"> <IKETransform IKE_Hash="MD5" IKE_DHGroup="1" IKE_Encryption="DES-CBC" IKE_KeyRefreshMinutes="480" IKE_AuthenticationMethod="Preshared_key"/> </IKEProtection> <IKETunnel IKE_TunnelName="P1" IKE_ProtectionRef="P1Pol" IKE_Flags_AutoStart="Yes" IKE_Flags_MakeRuleWithOptionalIP="No"> <IKELocalIdentity> <IPV4_Address Value="Local AIX Host IP"/> </IKELocalIdentity> <IKERemoteIdentity> <IPV4_Address Value="Remote Windows Server IP"/> </IKERemoteIdentity> </IKETunnel> <IKEPresharedKey Value="12345" Format="ASCII"> <IKEPresharedRemoteID> <PK_IPV4_Address Value="Remote Windows Server IP"/> </IKEPresharedRemoteID> </IKEPresharedKey> <IPSecProposal IPSec_ProposalName="P2Prop"> <IPSecAHProtocol AH_KeyRefreshKB="0" AH_Authentication="AH_MD5" AH_EncapsulationMode="Transport" AH_KeyRefreshMinutes="580"/> <IPSecESPProtocol ESP_Encryption="ESP_DES" ESP_KeyRefreshKB="0" ESP_Authentication="HMAC-MD5" ESP_EncapsulationMode="Transport" ESP_KeyRefreshMinutes="580"/> </IPSecProposal> <IPSecProtection IPSec_Role="Both" IPSec_KeyOverlap="10" IPSec_ProposalRefs="P2Prop " IPSec_ProtectionName="P2Pol" IPSec_InitiatorDHGroup="0" IPSec_ResponderDHGroup="NO_PFS GROUP_1 GROUP_2 GROUP_5" IPSec_Flags_UseLifeSize="No" IPSec_Flags_UseCommitBit="No" IPSec_ResponderKeyRefreshMaxKB="200" IPSec_ResponderKeyRefreshMinKB="1" IPSec_ResponderKeyRefreshMaxMinutes="220" IPSec_ResponderKeyRefreshMinMinutes="1"/> <IPSecTunnel IKE_TunnelName="P1" IPSec_TunnelName="P2" IPSec_ProtectionRef="P2Pol" IPSec_Flags_OnDemand="Yes" IPSec_Flags_AutoStart="Yes"> <IPSecLocalIdentity> <IPV4_Address Value="Local AIX Server IP"/> </IPSecLocalIdentity> <IPSecRemoteIdentity> <IPV4_Address Value="Remote Windows Server IP"/> </IPSecRemoteIdentity> </IPSecTunnel> </AIX_VPN>
Update new IPSec configuration in the IKE database
- We first need to remove the previous IPSec configuration in the IKE database, and then put the new configuration file in the ikedb.
- To remove the previous configuration, run the following command:
# ikedb -x P1_ITD database created successfully P2_ITD database created successfully P1_PREKEY database created successfully PROPOSAL_LIST database created successfully PROPOSAL database created successfully POLICY database created successfully GROUP database created successfully NDBM:/etc/ipsec/inet/DB/privkey
- To put the new configuration file in the database, run the following
# ikedb -p IPSECpolicy1
Check if all the three daemons (tmd, isakmpd and cpsd) are running. The tmd daemon takes care of the tunnel management, and the isakmpd daemon takes care of the IKE negotiation. If we are not using certificates for authentication, there is no need for the cpsd daemon to run.
To start the daemons, run the following command:
# startsrc -g ike 0513-059 The cpsd Subsystem has been started. Subsystem PID is 434304. 0513-059 The tmd Subsystem has been started. Subsystem PID is 315554. 0513-059 The isakmpd Subsystem has been started. Subsystem PID is 401504.
Run the following command to check if the daemons are started or not. If the daemon is started, the status of that daemon should be active.
# lssrc -g ike Subsystem Group PID Status cpsd ike 241894 active tmd ike 315550 active isakmpd ike 319648 active
Run the following command to check if any IPSec tunnel is active:
# ike cmd=list No tunnels match your request.
If you do not find the tunnels between the machines you actually intend to have the tunnel, then run the following command to activate the tunnels:
# ike cmd=activate Phase 2 tunnel 1 activate request initiated.
Now the ike cmd command should list the state of the tunnels for you.
# ike cmd=list Phase Tun Id Status Local Id Remote Id 1 1 Dormant 220.127.116.11 18.104.22.168 2 1 Dormant 22.214.171.124 126.96.36.199
We need to ping the remote host to activate the tunnels. One or two ping request may be denied until the tunnels become active. The requests will be successful from then on.
# ping 188.8.131.52 PING 184.108.40.206 (220.127.116.11): 56 data bytes ping: sendto: Permission denied ping: wrote 18.104.22.168 64 chars, ret=-1 64 bytes from 22.214.171.124: icmp_seq=1 ttl=255 time=0 ms 64 bytes from 126.96.36.199: icmp_seq=2 ttl=255 time=0 ms 64 bytes from 188.8.131.52: icmp_seq=3 ttl=255 time=0 ms
Now you have created a successful AIX to Windows IPSec tunnel that can be further used for secure communication over the network.
This article showcases the ability of AIX IPSec to work across heterogeneous environments. Similar to the Windows IPSec configuration reviewed in this article, you can try using other operating systems to communicate securely with AIX using IPSec. Doing so can provide greater security in an insecure public network with heterogeneous systems.
- Redbook: A Comprehensive Guide to Virtual Private Networks, Volume III: Cross-Platform Key and Policy Management
- An Illustrated Guide to IPSec
- pSeries and AIX Information Center
- Internet Information Services (IIS) 7.0 Administrator's Pocket Consultant by William R. Stanek
- Redbook: AIX 5L Version 5.2 Security Supplement
- Step-by-step guide to Internet Protocol Security (IPSec)
- Windows 2000 Server