IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 3
IKEv2 IPsec tunnels between AIX 6.1 or later versions and Windows 2012
IKEv2 tunnels using certificates and pre-shared keys
Content series:
This content is part # of # in the series: IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 3
This content is part of the series:IPsec tunnel configuration between IBM AIX and Microsoft Windows, Part 3
Stay tuned for additional content in this series.
Internet Protocol Security (IPsec) as its name suggests provides security at the Internet Protocol (IP) layer. This tutorial requires a basic understanding of what IPsec is and how it can be used to protect data over the network. You can refer to Knowledge Center or other resources like wiki to be acquainted with IPsec.
This tutorial discusses two different methods of establishing IPsec tunnels between AIX (6.1 / 7.1 / 7.2) and Windows 2012 systems. The methods involve pre-shared keys and certificates using IKEv2 between AIX and Windows systems. Table 1 provides a short description of the main topics covered in this tutorial and what they entail.
Table 1. Main topics covered
Content | Description |
---|---|
Terminologies and assumptions | This section provides a note on the important terms used in this tutorial and some assumptions on which the setup is configured. |
IKEv2 tunnels using certificates | This section explains the required Internet Key Exchange (IKE) XML file to be updated on AIX. It also has a detailed explanation for using the PowerShell CLI on Microsoft Windows 2012 for IKEv2. |
IKEv2 tunnels using pre-shared key | This section explains the required IKE XML file to be updated on AIX. Almost all the steps to be run on Windows 2012 are the same as mentioned in the "IKEv2 tunnels using certificates" section. The two methods differ by a single step on Windows. Only this single step has been highlighted in this section. |
Terminologies and assumptions
This section explains a few terms, such as initiator and responder, and highlights a few assumptions that this tutorial is based on.
- In this tutorial, for illustration purposes, we have mentioned the AIX system's IP as 1.1.1.1 and the Windows system's IP as 2.2.2.2. These need to be replaced with the appropriate IPs in your environment.
- Source and destination system matrix:
Table 2. Source and destination IPs
System Packet direction Source Destination On AIX Incoming 2.2.2.2 (Windows) 1.1.1.1 (AIX) On AIX Outgoing 1.1.1.1 (AIX) 2.2.2.2 (Windows) On Windows Incoming 1.1.1.1 (AIX) 2.2.2.2 (Windows) On Windows Outgoing 2.2.2.2 (Windows) 1.1.1.1 (AIX) The source is always the system that creates and sends a packet. The destination is always the system that receives it. This table (Table 2) needs to be read from left to right. For example, the first row is interpreted as follows:
'On AIX' system, when a packet is 'incoming', the source mentioned in the packet is '2.2.2.2 (Windows)' and the destination mentioned in this packet is '1.1.1.1 (AIX)'
- Initiator is the system that initiates a tunnel connection and the responder is the
system that responds to the initiator's request.
Either the Windows or the AIX system can be the initiator. You can activate the tunnels from Windows by pinging or communicating with AIX. Or, you can run the
ike cmd=activate
command on AIX and the tunnels will be active. If one of these methods don't work, try the other. - For this setup, you need to have the IPsec devices configured on AIX.
Running the
lsdev -Cc ipsec
command on AIX shows theipsec_v4
device as available. Else runsmitty ipsec4.
In the smitty panel:
- Select Start/Stop IP Security and press Enter.
- Select Start IP Security and press Enter.
- On the next screen, retain the default settings and press Enter.
- On the COMMAND STATUS screen, the message, ipsec_v4 Available, indicates successful configuration of the device.
IKEv2 tunnels between AIX and Windows using certificates
Before proceeding with the steps in this section, if you have assigned the IKEv1 policy to the Windows system, un-assign it. To un-assign the v1 policy, refer to the steps shown in Figure 40 in IKEv1 IPsec tunnels between AIX 6.1 or later versions and Windows 2012
If a policy is assigned, right-clicking the policy will show Unassign instead of Assign. Click Unassign to unassign the policy.
Refer the tutorial, Generating certificates in AIX and importing certificates to Windows for IKE IPsec tunnels for creating the certificates on AIX and importing to the Windows operating system before proceeding with the following steps.
The following XML file needs to be created on the AIX system. Let us name it AIX-Windows-Certificates-IKEv2.xml. Add this XML to the IKE database on AIX using the following commands:
/usr/sbin/ikedb -x /usr/sbin/ikedb -p AIX-Windows-Certificates-IKEv2.xml <?xml version="1.0"?> <AIX_VPN Version="2.0"> <IKEProtection IKE_Role="Both" IKE_Version="2" IKE_XCHGMode="Main" IKE_KeyOverlap="10" IKE_Flags_UseCRL="No" IKE_ProtectionName="P1Pol" IKE_ResponderKeyRefreshMaxKB="200" IKE_ResponderKeyRefreshMinKB="1" IKE_ResponderKeyRefreshMaxMinutes="1440" IKE_ResponderKeyRefreshMinMinutes="1"> <IKETransform IKE_Encryption="AES-CBC-256" IKE_PRF="PRF_SHA2_256" IKE_Hash="SHA2_256" IKE_DHGroup="2" IKE_AuthenticationMethod="RSA_signatures"/> </IKEProtection> <IKETunnel IKE_TunnelName="P1" IKE_ProtectionRef="P1Pol" IKE_Flags_AutoStart="No" IKE_Flags_MakeRuleWithOptionalIP="Yes"> <IKELocalIdentity> <ASN1_DN Value="/C=IN/ST=KA/L=BA/O=IBM/OU=ISL/CN=test2"> <IPV4_Address Value="1.1.1.1"/> </ASN1_DN> </IKELocalIdentity> <IKERemoteIdentity> <ASN1_DN Value="/C=IN/ST=KA/L=BA/O=IBM/OU=ISL/CN=test1"> <IPV4_Address Value="2.2.2.2"/> </ASN1_DN> </IKERemoteIdentity> </IKETunnel> <IPSecProposal IPSec_ProposalName="P2Prop"> <IPSecESPProtocol ESP_Encryption="ESP_AES_256" ESP_KeyRefreshKB="0" ESP_Authentication="HMAC-SHA" ESP_ExtendedSeqNum="0" ESP_EncapsulationMode="Transport" ESP_KeyRefreshMinutes="30"/> </IPSecProposal> <IPSecProtection IPSec_Role="Both" IPSec_KeyOverlap="10" IPSec_ProposalRefs="P2Prop " IPSec_ProtectionName="P2Pol" IPSec_InitiatorDHGroup="0" IPSec_ResponderDHGroup="NO_PFS" IPSec_Flags_UseLifeSize="No" IPSec_Flags_UseCommitBit="No" IPSec_ResponderKeyRefreshMaxKB="200" IPSec_ResponderKeyRefreshMinKB="1" IPSec_ResponderKeyRefreshMaxMinutes="43200" IPSec_ResponderKeyRefreshMinMinutes="1"/> <IPSecTunnel IKE_TunnelName="P1" IPSec_TunnelName="P2" IPSec_ProtectionRef="P2Pol" IPSec_Flags_OnDemand="No" IPSec_Flags_AutoStart="No"> <IPSecLocalIdentity> <IPV4_Address_Range To_IPAddr="1.1.1.1" From_IPAddr="1.1.1.1"/> </IPSecLocalIdentity> <IPSecRemoteIdentity> <IPV4_Address_Range To_IPAddr="2.2.2.2" From_IPAddr="2.2.2.2"/> </IPSecRemoteIdentity> </IPSecTunnel> </AIX_VPN>
To keep it simple, let's start only the IKEv2 daemon using the following commands:
stopsrc -g ike startsrc -s tmd ; startsrc -s ikev2d; startsrc -s cpsd
When we start only ikev2d
(IKEv2 daemon) we don't need to start the
isakmpd
(IKEv1) or iked
(broker) daemons. Only the
tmd
daemon is required.
You need to perform the following steps on Widows PowerShell on a Windows 2012 system to create the IKEv2 policy with certificates.
This is an example set of commands and must not be treated as the standard way to implement
IKEv2 on Windows. You can change the names and attributes as per your requirement. Also, we
have used PersistentStore
as our PolicyStoreSource
and have used
Local
for PolicyStoreSourceType
on Windows 2012.
PS C:\ > $IPsAP = New-NetIPsecAuthProposal -Machine -Cert -Authority "C=IN,
O=IBM, CN=ipsecroot" -AuthorityType Root
PS C:\ > $IPsP1AS = New-NetIpsecPhase1Authset -DisplayName "Phase1 Auth Set"
-proposal $IPsAP
Figure 1. get-NetIPsecPhase1AuthSet output

PS C:\ > $IPsMMCP = New-NetIPsecMainModeCryptoProposal -Encryption AES256
-Hash SHA256 -KeyExchange DH2
PS C:\> $IPsMMCS = New-NetIPsecMainModeCryptoSet -DisplayName "Main Mode
crypto set" -proposal $IPsMMCP
Figure 2. get-NetIPsecMainModeCryptoSet output

PS C:\ > New-NetIPsecMainModeRule -DisplayName "Main Mode Rule"
-MainModeCryptoSet $IPsMMCS.Name -Phase1AuthSet $IPsP1AS.Name
Figure 3. New-NetIPsecMainModeRule output

PS C:\ > $IPsQMCP = New-NetIPsecQuickModeCryptoProposal -Encapsulation ESP
-ESPHash SHA1 -Encryption AES256
PS C:\ > $IPsQMCS = New-NetIPsecQuickModeCryptoSet -DisplayName "IPsec quick
Mode crypto set" -proposal $IPsQMCP
We can run Get-NetIPsecMainModeCryptoSet
to check the output from the
previous cmdlets.
Figure 4. get-NetIPsecQuickModeCryptoSet output

PS C:\ > New-NetIpsecRule -DisplayName "Ipsec Rule" -localaddress 2.2.2.2 -remoteaddress 1.1.1.1 -phase1AuthSet $IPsP1AS.InstanceID -requireAuthorization $true -Inboundsecurity require -Outboundsecurity require -KeyModule IKEv2 -Quickmodecryptoset $IPsQMCS.Name
Figure 5. New-NetIPsecRule output

PS C:\ > Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled
True
Note that when you run the Set-NetFirewallProfile
command, other systems
without the IPsec rule might not be able to connect to the Windows system based on the
default options in Set-NetFireWallProfile
. You need to check the command's
options on the Microsoft official website to override the default options if you face any
problems.
If you want to undo the above configuration, start by removing the rules which were run the
last and move backwards. That is, proceed by removing the last rule using the
remove
-
NetIpsecRule
command followed by remove-NetIPsecQuickModeCryptoSet
,
remove-NetIPsecMainModeRule
,
remove-NetIPsecMainModeCryptoSet
and finally
remove-NetIpsecPhase1AuthSet
.
To remove the IPsec rule created, run the following cmdlet:
remove-NetIpsecRule -displayName "Ipsec Rule"
To remove a QuickModeCrypto
set, use the following cmdlet:
remove-NetIPsecQuickModeCryptoSet -displayName "IPsec quick Mode crypto set"
To remove MainModeRule
use the following cmdlet:
remove-NetIPsecMainModeRule -displayName "Main Mode Rule"
To delete MainModeCryptoSet
, use the following cmdlet:
remove-NetIPsecMainModeCryptoSet -displayName "Main Mode Crypto set"
To delete this phase1AuthSet
, use the following cmdlet:
remove-NetIpsecPhase1AuthSet -displayName "Phase1 Auth set"
IKEv2 tunnels between AIX and Windows using pre-shared keys
Before proceeding with the steps in this section, if you have assigned the IKEv1 policy to the Windows system, unassign it. To unassign the v1 policy, refer to the steps shown in Figure 40 in IKEv1 IPsec tunnels between AIX 6.1 or later versions and Windows 2012 . If a policy is assigned, right-clicking the policy shows Unassign instead of Assign. Click Unassign to unassign the policy.
Else, if you have tried the IKEv2 steps mentioned earlier, run the remove-
commands cited in the previous section before proceeding further. Next, run the IKEv2
PowerShell commands afresh with the required changes mentioned earlier.
The following XML file needs to be created on the AIX system. Let us name it AIX-Windows-PreShared-IKEv2.xml. Add this XML file to the IKE database on AIX using the following commands:
/usr/sbin/ikedb -x /usr/sbin/ikedb -p AIX-Windows-PreShared-IKEv2.xml
<?xml version="1.0"?> <AIX_VPN Version="2.0"> <IKEProtection IKE_Role="Both" IKE_Version="2" IKE_XCHGMode="Main" IKE_KeyOverlap="10" IKE_Flags_UseCRL="No" IKE_ProtectionName="P1Pol" IKE_ResponderKeyRefreshMaxKB="200" IKE_ResponderKeyRefreshMinKB="1" IKE_ResponderKeyRefreshMaxMinutes="1440" IKE_ResponderKeyRefreshMinMinutes="1"> <IKETransform IKE_Encryption="AES-CBC-256" IKE_PRF="PRF_SHA2_256" IKE_Hash="SHA2_256" IKE_DHGroup="2" IKE_AuthenticationMethod="Preshared_key"/> </IKEProtection> <IKETunnel IKE_TunnelName="P1" IKE_ProtectionRef="P1Pol" IKE_Flags_AutoStart="No" IKE_Flags_MakeRuleWithOptionalIP="Yes"> <IKELocalIdentity> <IPV4_Address Value="1.1.1.1"/> </IKELocalIdentity> <IKERemoteIdentity> <IPV4_Address Value="2.2.2.2"/> </IKERemoteIdentity> </IKETunnel> <IKEPresharedKey Value="12345" Format="ASCII"> <IKEPresharedRemoteID> <PK_IPV4_Address Value="2.2.2.2"/> </IKEPresharedRemoteID> </IKEPresharedKey> <IPSecProposal IPSec_ProposalName="P2Prop"> <IPSecESPProtocol ESP_Encryption="ESP_AES_256" ESP_KeyRefreshKB="0" ESP_Authentication="HMAC-SHA" ESP_ExtendedSeqNum="0" ESP_EncapsulationMode="Transport" ESP_KeyRefreshMinutes="480"/> </IPSecProposal> <IPSecProtection IPSec_Role="Both" IPSec_KeyOverlap="10" IPSec_ProposalRefs="P2Prop " IPSec_ProtectionName="P2Pol" IPSec_InitiatorDHGroup="5" IPSec_ResponderDHGroup="GROUP_5" IPSec_Flags_UseLifeSize="No" IPSec_Flags_UseCommitBit="No" IPSec_ResponderKeyRefreshMaxKB="200" IPSec_ResponderKeyRefreshMinKB="1" IPSec_ResponderKeyRefreshMaxMinutes="43200" IPSec_ResponderKeyRefreshMinMinutes="1"/> <IPSecTunnel IKE_TunnelName="P1" IPSec_TunnelName="P2" IPSec_ProtectionRef="P2Pol" IPSec_Flags_OnDemand="No" IPSec_Flags_AutoStart="No"> <IPSecLocalIdentity> <IPV4_Address_Range To_IPAddr="1.1.1.1" From_IPAddr="1.1.1.1"/> </IPSecLocalIdentity> <IPSecRemoteIdentity> <IPV4_Address_Range To_IPAddr="2.2.2.2" From_IPAddr="2.2.2.2"/> </IPSecRemoteIdentity> </IPSecTunnel> </AIX_VPN>
To keep it simple, let's start only the IKEv2 daemon using the following commands:
stopsrc -g ike startsrc -s tmd ; startsrc -s ikev2d
When we start only ikev2d
(IKEv2 daemon) we don't need to start the
isakmpd
(IKEv1) or iked
(broker) daemons. Only the
tmd
daemon is required.
There is a difference only in one step between the PowerShell cmdlets for IKEv2 tunnels
with pre-shared keys and certificates. Instead of certificates, in the first step in the
"IKEv2 tunnels between AIX and Windows using
certificates" section, we will use pre-shared keys in
New-NetIPsecAuthProposal
as follows:
PS C:\ > $IPsAP = New-NetIPsecAuthProposal -Machine -PreSharedKey
"12345"
All other PowerShell steps that follow
New-NetIpsecAuthProposal
in the "IKEv2 tunnels between AIX and Windows using
certificates" section are the same for the pre-shared key configuration as well.
We will now see pre-shared key data in the output of
get-NetIPsecPhase1AuthSet
:
Figure 6. get-NetIPsecPhase1AuthSet output

Summary
This tutorial explained how to establish IKEv2 tunnels between AIX 6.1/ 7.1/ 7.2 and Windows 2012 systems using certificates and pre-shared keys. The following tutorials explains how to establish tunnels using IKEv1 between AIX and Windows operating system:
IKEv1 IPsec tunnels between AIX 6.1 or later versions and Windows 2012
Part 2 and Part 3, together, explain four different ways to establish tunnels between AIX 6.1/ 7.1/ 7.2 and Windows 2012 systems. These four methods are setting up IKEv1 and IKEv2 tunnels using pre-shared keys and certificates between AIX and Windows.
The configuration steps for Windows (explained in this tutorial) were tried in the lab and worked for the test team. These steps are not endorsed by Microsoft nor by IBM. All the steps that were run on PowerShell were developed using the Microsoft documents available online. You can consider the steps mentioned in this tutorial as a guidance to get you started. There is always light at the end of the tunnel. So, enjoy configuring tunnels!