IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 2
IKEv1 IPsec tunnels between AIX 6.1 or later versions and Windows 2012
IKEv1 tunnels using certificates and pre-shared keys
Content series:
This content is part # of # in the series: IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 2
This content is part of the series:IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 2
Stay tuned for additional content in this series.
Internet Protocol Security (IPsec) as its name suggests provides security at the Internet Protocol (IP) layer. This tutorial requires a basic understanding of what IPsec is and how it can be used to protect data over the network. You can refer to Knowledge Center or other resources such as wiki to get acquainted with IPsec.
This tutorial discusses two different methods of establishing IPsec tunnels between IBM® AIX® (6.1 / 7.1 / 7.2) and Microsoft Windows 2012 systems. The methods involve pre-shared keys and certificates using IKEv1 between AIX and Windows systems. Table 1 provides a short description of the main topics covered in this tutorial and what they entail.
Table 1. Main topics covered
Content | Description |
---|---|
Terminologies and assumptions | This section provides a note on the important terms used in this tutorial and some assumptions on which the setup is configured. |
IKEv1 tunnels using pre-shared keys | This section explains the required Internet Key Exchange (IKE) XML file to be updated on AIX. It also has a detailed explanation for using GUI on Windows 2012 for IKEv1. |
IKEv1 tunnels using certificates | This section explains the required IKE XML file to be updated on AIX. Almost all the steps to be run on Windows 2012 are the same as mentioned in the "IKEv1 tunnels using pre-shared key" section. The two methods differ by a single step on Windows. Only this single step has been highlighted in this section. |
Terminologies and assumptions
This section explains a few terms, such as initiator and responder, and highlights a few assumptions that this tutorial is based on.
- In this tutorial, for illustration purposes, we have mentioned the AIX system's IP as 1.1.1.1 and the Windows system's IP as 2.2.2.2. These need to be replaced with the appropriate IPs in your environment.
- Source and destination system matrix:
Table 2. Source and destination IPs
System Packet direction Source Destination On AIX Incoming 2.2.2.2 (Windows) 1.1.1.1 (AIX) On AIX Outgoing 1.1.1.1 (AIX) 2.2.2.2 (Windows) On Windows Incoming 1.1.1.1 (AIX) 2.2.2.2 (Windows) On Windows Outgoing 2.2.2.2 (Windows) 1.1.1.1 (AIX) The source is always the system that creates and sends a packet. The destination is always the system that receives it. This table (Table 2) needs to be read from left to right. For example, the first row is interpreted as follows:
'On AIX' system, when a packet is 'incoming', the source mentioned in the packet is '2.2.2.2 (Windows)' and the destination mentioned in this packet is '1.1.1.1 (AIX)'
- Initiator is the system that initiates a tunnel connection and the
responder is the system that responds to the initiator's request.
Either the Windows or the AIX system can be the initiator. You can activate the tunnels from Windows by pinging or communicating with AIX. Or, you can run the
ike cmd=activate
command on AIX and the tunnels will be active. If one of these methods don't work, try the other. - For this setup, you need to have the IPsec devices configured on AIX.
Running the
lsdev -Cc ipsec
command on AIX shows theipsec_v4
device as available. Else runsmitty ipsec4
.In the smitty panel:
- Select Start/Stop IP Security and press Enter.
- Select Start IP Security and press Enter.
- On the next screen, retain the default settings and press Enter.
- On the COMMAND STATUS screen, the message, ipsec_v4 Available, indicates successful configuration of the device.
IKEv1 tunnels between AIX and Windows using pre-shared keys
The following XML file needs to be created on the AIX system. Let's name it AIX-Windows-PreShared-IKEv1.xml. Add this XML file to the IKE database on AIX using the following command:
/usr/sbin/ikedb -x /usr/sbin/ikedb -p AIX-Windows-PreShared-IKEv1.xml <?xml version="1.0"?> <AIX_VPN Version="2.1"> <IKEProtection IKE_Role="Both" IKE_Version="1" IKE_XCHGMode="Main" IKE_KeyOverlap="10" IKE_Flags_UseCRL="No" IKE_ProtectionName="P1Pol" IKE_ResponderKeyRefreshMaxKB="200" IKE_ResponderKeyRefreshMinKB="1" IKE_ResponderKeyRefreshMaxMinutes="1440" IKE_ResponderKeyRefreshMinMinutes="60"> <IKETransform IKE_Encryption="3DES-CBC" IKE_Hash="SHA" IKE_DHGroup="1" IKE_AuthenticationMethod="Preshared_key" IKE_KeyRefreshMinutes="240"/> </IKEProtection> <IKETunnel IKE_TunnelName="P1" IKE_ProtectionRef="P1Pol" IKE_Flags_AutoStart="Yes" IKE_Flags_MakeRuleWithOptionalIP="No"> <IKELocalIdentity> <IPV4_Address Value="1.1.1.1"/> </IKELocalIdentity> <IKERemoteIdentity> <IPV4_Address Value="2.2.2.2"/> </IKERemoteIdentity> </IKETunnel> <IKEPresharedKey Value="12345" Format="ASCII"> <IKEPresharedRemoteID> <PK_IPV4_Address Value="2.2.2.2"/> </IKEPresharedRemoteID> </IKEPresharedKey> <IPSecProposal IPSec_ProposalName="P2Prop"> <IPSecESPProtocol ESP_Encryption="ESP_3DES" ESP_KeyRefreshKB="0" ESP_Authentication="HMAC-SHA" ESP_ExtendedSeqNum="0" ESP_EncapsulationMode="Transport" ESP_KeyRefreshMinutes="30"/> </IPSecProposal> <IPSecProtection IPSec_Role="Both" IPSec_KeyOverlap="10" IPSec_ProposalRefs="P2Prop " IPSec_ProtectionName="P2Pol" IPSec_InitiatorDHGroup="0" IPSec_ResponderDHGroup="NO_PFS" IPSec_Flags_UseLifeSize="No" IPSec_Flags_UseCommitBit="No" IPSec_ResponderKeyRefreshMaxKB="200" IPSec_ResponderKeyRefreshMinKB="1" IPSec_ResponderKeyRefreshMaxMinutes="43200" IPSec_ResponderKeyRefreshMinMinutes="30"/> <IPSecTunnel IKE_TunnelName="P1" IPSec_TunnelName="P2" IPSec_ProtectionRef="P2Pol" IPSec_Flags_OnDemand="No" IPSec_Flags_AutoStart="Yes"> <IPSecLocalIdentity Port="0" EndPort="65535" Protocol="0"> <IPV4_Address_Range To_IPAddr="1.1.1.1" From_IPAddr="1.1.1.1"/> </IPSecLocalIdentity> <IPSecRemoteIdentity Port="0" EndPort="65535" Protocol="0"> <IPV4_Address_Range To_IPAddr="2.2.2.2" From_IPAddr="2.2.2.2"/> </IPSecRemoteIdentity> </IPSecTunnel> </AIX_VPN>
To keep it simple, let's start only the IKEv1 daemon using the following commands:
stopsrc -g ike startsrc -s tmd ; startsrc -s isakmpd
When we start only isakmpd
(IKEv1 daemon), we don't need to start
the ikev2d
(IKEv2) or iked
(broker) daemons. Only
the tmd
daemon (tunnel manager daemon) is required.
You need to perform the following steps on the Windows 2012 system to create the IKEv1 IPsec policy. The overall tasks can be divided into the following five high-level steps:
- Add IPsec snap-in to the Microsoft Management Console (MMC)
- Create the IPsec policy.
- Create the IPsec filter rules.
- Define IP tunneling attributes.
- Define authentication methods.
- Define filter actions.
- Specify other tunnel-related settings.
- Assign the IPsec policy.
Add IPsec snap-in to the MMC
Perform the following steps to add an IPsec snap-in to the MMC:
- Open the MMC.
- Click File -> Add/Remove
Snap-in.
Figure 1. MMC File menu option
- From the Available snap-ins list, select IP Security Policy
Manager and click Add. Then, click
OK.
Figure 2. Adding the IP Security Policy Management snap-in
- In the Select Computer or Domain dialog box, select Local
computer and click Finish.
Figure 3. Selecting the computer or domain to manage snap-in
Clicking Finish takes you back to the Add or Remove Snap-ins dialog box. Click OK.
Create the IPsec policy
Perform the following steps to create the IPsec policy:
- In the main console, right-click IP Security Policies on Local
Computer and then click Create IP Security
Policy.
Figure 4. Right-click menu options for IPsecurity policies snap-in
Figure 5. Welcome page of IP Security Policy Wizard
- On the welcome page of the IP Security Policy Wizard, click
Next. On the IP Security Policy Name page,
click Next to create a policy named
Policy1.
Figure 6. IP security policy naming
- On the Request for Secure Communication page, click
Next.
Figure 7. IP security policy wizard
- On the wizard completion page, select the Edit
properties check box and click Finish.
Figure 8. IP Security Policy Wizard completion
Create the IPsec filter rules
Perform the following steps to create the IPsec filter rules:
- To set the policy properties, select the Use Add Wizard
check box at the lower-right corner and then click
Add.
Figure 9. Adding policy properties to the newly created policy
- On the welcome page of the Security Rule Wizard, click
Next.
Figure 10. Security Rule Wizard welcome page
- Define the IP tunneling attributes.
- Select the This rule does not specify a
tunnel option and click
Next.
Figure 11. Tunnel Endpoint page
- On the Network Type page, select All
network connections and click
Next.
Figure 12. Selecting the network type
- On the IP Filter List page, click
Add.
Figure 13. Adding IP filters
- You can give a meaningful name to the filter
list. In this example, we have entered
Policy 1 IP Filter List 1. Note
that Use Add Wizard check box is
selected and click Add.
Figure 14. Creating an IP filter policy
- On the welcome page of the IP Filter
Wizard, click Next.
Figure 15. IP Filter Wizard welcome page
- On the IP Filter Description and Mirrored
property page, select the Mirrored. Match
packets with the exact opposite source and
destination option to apply the same rule
on both the incoming and outgoing packets on
Windows. Then click Next.
Figure 16. IP Filter Description and Mirrored property page
- On the IP Traffic Source page, from the Source
address drop-down list, select A specific
IP Address or Subnet.
Figure 17. IP traffic source selection
Also, provide the source IP below the drop-down list (the partner system's IP = AIX system's IP) and click Next.
- On the IP Traffic Destination page, from the
Destination address drop-down list, select
A specific IP Address or subnet,
add the required destination IP, and click
Next.
Figure 18. IP traffic destination selection
- On the IP Protocol Type page, in this example,
we are selecting Any as the
protocol type. You can select the required
protocol as per your needs. After that, click
Next.
Figure 19. IP traffic protocol type
- On the wizard completion page, click
Finish.
Figure 20. IP Filter Wizard completion
- Notice that you are directed to the IP Filter
List dialog box. Click OK in
dialog box.
Figure 21. IP Filter List dialog box
- Select the This rule does not specify a
tunnel option and click
Next.
- Define IP filter actions.
- On the IP Filter List page, select
Policy 1 IP Filter List 1 and
click Next.
Figure 22. IP Filter List page
- On the Filter Action page, if
you see an existing filter action, you can select
it and click Edit. Else you can
add a new filter action if you do not want to
modify an existing one. In this example, we are
adding a new one. Notice that the Use Add
Wizard check box is checked. Click
Add.
Figure 23. Filter Action page
- As Use Add Wizard is selected,
the IP Security Filter Action
Wizard opens. Click Next
on the welcome page.
Figure 24. IP Security Filter Action Wizard welcome page
- We will change the default name New
filter action to Policy 1 filter
action 1. Then, click
Next.
Figure 25. Filter Action Name page
- On the Filter Action General Options page,
select Negotiate security and
click Next.
Figure 26. Filter Action General Options page
- On the Communicating with computers that do not
support IPsec page, select Do not allow
unsecured communication and click
Next.
Figure 27. IP filter action on communicating with IPsec unaware systems
- On the IP Traffic Security page, select
Integrity and encryption and
click Next.
Figure 28. IP Traffic Security page
- On the wizard completion page, click
Finish.
Figure 29. IP filter action completion page
- Notice that you are directed to the Filter
Action page. On this page, select the new filter
action, Policy 1 filter action 1.
Notice that the Use Add Wizard
check box is selected and click
Next.
Figure 30. Back at the Filter Action page
- On the IP Filter List page, select
Policy 1 IP Filter List 1 and
click Next.
- Define authentication methods.
- On the Authentication Method page, select
Use this string to protect the key
exchange (pre-shared
key) and click Next.
Note that this key is the same one that is
mentioned in the XML file updated on AIX.
Figure 31. Selecting the authentication method
- On the Security Rule Wizard completion page,
click Finish.
Figure 32. Security rule completion page
- In the Policy1 Properties
dialog box, click the
General tab.
Figure 33. Policy1 Properties dialog box
- On the Authentication Method page, select
Use this string to protect the key
exchange (pre-shared
key) and click Next.
Note that this key is the same one that is
mentioned in the XML file updated on AIX.
- Specify other tunnel-related settings.
- On the General tab, click
Settings to specify additional
settings.
Figure 34. General tab of Policy1 properties
- In the Key Exchange Settings dialog box, set the
time after which a new key must be regenerated.
Then click Methods.
Figure 35. Policy properties: Key exchange settings
- In the Key Exchange Security Methods dialog box,
select the rule that is displayed and click
Edit.
You can change the algorithms used for encryption and integrity checking and you can set the Diffie-Hellman group level here. Make sure that the IKE security algorithms match the ones mentioned in the XML file on AIX. The settings we use are the ones shown in the following screen captures.
After the required algorithms have been chosen, click OK and proceed to the next steps.
Figure 36. Key exchange settings: Key Exchange Security Methods dialog box
Figure 37. Key Exchange Security Methods dialog box
- In the Key Exchange Security Methods dialog box,
click OK. Also, in the Policy1
Properties dialog box, click OK.
Figure 38. Back to Policy1 Properties
- On the General tab, click
Settings to specify additional
settings.
- Assign the IPsec policy.
- After the policy is created, notice that it is
not initially assigned to the system.
Figure 39. Console root with IP security snap-in
- Right-click Policy1 and click
Assign.
Figure 40. Assigning the IP security Policy
- Notice that the Policy Assigned state now
changes to Yes from
No.
Figure 41. IP security Policy assigned
- After the policy is created, notice that it is
not initially assigned to the system.
- Define the IP tunneling attributes.
IKEv1 tunnels between AIX and Windows using certificates
Follow the certificate generation steps on the AIX system and the certificate import steps on Windows explained in their respective sections earlier.
Refer to the tutorial Generating certificates in AIX and importing certificates to Windows for IKE IPsec tunnels for creating the certificates on AIX and importing to the Windows operating system before proceeding with the following steps.
You need to create the following XML file in the AIX system. Let us name it AIX-Windows-Certificates-IKEv1.xml. Add this XML to the IKE database on AIX using the following commands:
/usr/sbin/ikedb -x /usr/sbin/ikedb -p AIX-Windows-Certificates-IKEv1.xml
<?xml version="1.0"?> <AIX_VPN Version="2.1"> <IKEProtection IKE_Role="Both" IKE_Version="1" IKE_XCHGMode="Main" IKE_KeyOverlap="10" IKE_Flags_UseCRL="No" IKE_ProtectionName="P1Pol" IKE_ResponderKeyRefreshMaxKB="200" IKE_ResponderKeyRefreshMinKB="1" IKE_ResponderKeyRefreshMaxMinutes="1440" IKE_ResponderKeyRefreshMinMinutes="60"> <IKETransform IKE_Encryption="3DES-CBC" IKE_Hash="SHA" IKE_DHGroup="1" IKE_AuthenticationMethod="RSA_signatures" IKE_KeyRefreshMinutes="240"/> </IKEProtection> <IKETunnel IKE_TunnelName="P1" IKE_ProtectionRef="P1Pol" IKE_Flags_AutoStart="Yes" IKE_Flags_MakeRuleWithOptionalIP="Yes"> <IKELocalIdentity> <ASN1_DN Value="/C=IN/ST=KA/L=BA/O=IBM/OU=ISL/CN=test2"> <IPV4_Address Value="1.1.1.1"/> </ASN1_DN> </IKELocalIdentity> <IKERemoteIdentity> <ASN1_DN Value="/C=IN/ST=KA/L=BA/O=IBM/OU=ISL/CN=test1"> <IPV4_Address Value="2.2.2.2"/> </ASN1_DN> </IKERemoteIdentity> </IKETunnel> <IPSecProposal IPSec_ProposalName="P2Prop"> <IPSecESPProtocol ESP_Encryption="ESP_3DES" ESP_KeyRefreshKB="0" ESP_Authentication="HMAC-SHA" ESP_ExtendedSeqNum="0" ESP_EncapsulationMode="Transport" ESP_KeyRefreshMinutes="30"/> </IPSecProposal> <IPSecProtection IPSec_Role="Both" IPSec_KeyOverlap="10" IPSec_ProposalRefs="P2Prop " IPSec_ProtectionName="P2Pol" IPSec_InitiatorDHGroup="0" IPSec_ResponderDHGroup="NO_PFS" IPSec_Flags_UseLifeSize="No" IPSec_Flags_UseCommitBit="No" IPSec_ResponderKeyRefreshMaxKB="200" IPSec_ResponderKeyRefreshMinKB="1" IPSec_ResponderKeyRefreshMaxMinutes="43200" IPSec_ResponderKeyRefreshMinMinutes="30"/> <IPSecTunnel IKE_TunnelName="P1" IPSec_TunnelName="P2" IPSec_ProtectionRef="P2Pol" IPSec_Flags_OnDemand="No" IPSec_Flags_AutoStart="Yes"> <IPSecLocalIdentity Port="0" EndPort="65535" Protocol="0"> <IPV4_Address_Range To_IPAddr="1.1.1.1" From_IPAddr="1.1.1.1"/> </IPSecLocalIdentity> <IPSecRemoteIdentity Port="0" EndPort="65535" Protocol="0"> <IPV4_Address_Range To_IPAddr="2.2.2.2" From_IPAddr="2.2.2.2"/> </IPSecRemoteIdentity> </IPSecTunnel> </AIX_VPN>
To keep it simple, let’s start only the IKEv1 daemon using the following commands:
stopsrc -g ike startsrc -s tmd ; startsrc -s isakmpd ; startsrc -s cpsd
When we start only isakmpd
(IKEv1 daemon), we don’t need to start
the ikev2d
(IKEv2) or iked
(broker) daemons. Only
the tmd
daemon is required.
Perform all the steps mentioned for the Windows system in the section, IKEv1 tunnels between AIX and windows using pre-shared keys except the steps showed in Figure 31. Instead of using pre-shared specify the settings as shown in Figure 42 for the certificates.
Figure 42. Authentication Method page in Security Rule Wizard

Summary
This tutorial explained how to establish IKEv1 tunnels between AIX 6.1/ 7.1/ 7.2 and Windows 2012 systems using certificates and pre-shared keys. The following tutorial explains how to establish tunnels using IKEv2 between AIX and Windows Operating system:
IKEv2 IPsec tunnels between AIX 6.1 or later versions and Windows 2012
Part 2 and part 3 in this series, together, explain four different ways to establish tunnels between AIX 6.1/ 7.1/ 7.2 and Windows 2012 systems. These four methods are setting up IKEv1 and IKEv2 tunnels using pre-shared keys and certificates between AIX and Windows.
The configuration steps for Windows (explained in this tutorial) were tried in the lab and worked for the test team. These steps are not endorsed by Microsoft nor by IBM. You can consider the steps mentioned in this tutorial as a guidance to get you started. There is always light at the end of the tunnel. So, enjoy configuring tunnels!
Related topics
- For tunnel creation with IKEv1 between older versions of AIX and older Windows systems, refer Heterogeneous IPSec solution between AIX and Windows.
- Refer RFC for IKEv1.