Exploring IKEv2 with ECDSA certificate in IBM AIX

Internet security is a major concern now-a-days. Internet Protocol Security (IPSec) is a security protocol suite which works on the L3 layer of network and provides a strong security with the help of different types of authentication and encryption mechanisms. This article discusses about the usage of the Elliptic Curve Digital Signature Algorithm (ECDSA) certificate based authentication mechanism for Internet Key Exchange Version 2 (IKEv2).


Souradeep Chakrabarti (souradch@in.ibm.com), Software Engineer, IBM

Souradeep Chakrabarti's PhotoSouradeep Chakrabarti works as a software engineer in the IBM AIX L3 team at Bangalore, India. He has been working on AIX, IPSec, and TCP/IP components on IBM Power Systems for more than 1 year at IBM India System and Technology Lab.

23 July 2014


IPSec is a security protocol suite, which works on the L3 layer of network. IPSec is independent of applications. So, developers do not need to modify the application to use IPSec.

IPSec has the following three sub protocols:

  • Authentication Header (AH)
  • Encapsulating Security Payload (ESP)
  • Internet Key Exchange (IKE) protocol

The AH protocol assures integrity protection, the ESP protocol provides encryption services and optional integrity protection, and the IKE protocol allows communicating entities to derive session keys for a secure communication using a series of message exchanges.

The AH protocol defines one mandatory and several optional transforms used to provide authentication, integrity, and replay detection. The ESP protocol defines one mandatory and many optional transforms that are used to provide data confidentiality. (See Figure 1 and Figure 2)

Figure 1. AH header format
Figure 2. ESP header format

IKE exists in two versions IKEv1 and IKEv2. Although IKEv1 is flexible and contains diverse options, its complexity remains a major problem for a widespread implementation, and that is why constant evolution of the first version was inevitable. IKEv2 tries to make understanding easier than IKEv1; thanks to the number of exchange messages and the renewed choices of the encryption and authentication algorithms. Both have pros and cons.

Phase I is when the authentication of IKE and the negotiation of IKE Security Association (SA) is done. Phase II, also known as CREATE_CHILD_SA in IKEv2, is when the IPSec SA parameters are negotiated.

The SA is simply a contract between two entities (aixtcp1 and aixtcp02 in this example) to provide a minimum set of services. It can be bidirectional (as in phase I) or unidirectional (as in phase II). If unidirectional, which is often the case, we might need two phase II SAs to complete one communication. SA is a set of proposals. A proposal can be thought as a set of protocols and a protocol is, in turn, is a set of transforms. A transform is a set of algorithms. Figure 3 shows how an end-to-end security is used after an IPSec tunnel is created.

Figure 3. End-to-end security using IPSec

The purpose of phase I is to generate the shared secret from which other keys will be computed and authenticate the communicating peers.

Phase II is used to create an IPSec SA and to generate new keys. In IKEv2, this SA is denoted as Child-SA, which is created as a result of the Create-Child-SA request. Currently, there are two digital signature methods defined for use within Phase 1 of IKEv2: RSA signatures and ECDSA. Now, how messages are exchanged during phase I and phase II of IKEv2 are depicted in Figure 4.

Figure 4. IKEv2 exchange (phase I and phase II)

In Figure 4, the notations are used for the payloads contained in the message. Refer to Table 1 for notations.

Table 1. Notations used in Figure 4
AUTH Authentication
CERT Certificate
CERTREQ Certificate request
CP Configuration
D Delete
E Encrypted
HDR IKE Header
IDi Identification - Initiator
IDr Identification - Responder
KE Key Exchange
Ni, Nr Nonce
N Notify
SA Security Association
TSi Traffic Selector - Initiator
TSr Traffic Selector - Responder
V Vendor ID
EAP Extensible Authentication

Why ECDSA signatures

ECDSA is the elliptic curve analog of the Digital Signature Standard (DSS). [RFC3279] describes ways to carry ECDSA keys in X.509 certificates. Similar to DSA, ECDSA also incorporates the use of a hash function. Currently, only one type of hash function is defined for use with ECDSA and that is SHA-1 message digest algorithm [FIPS-180-1].

Both, ECDSA and Rivest-Shamir-Adleman (RSA) are used in public key cryptography. ECDSA is mathematically more complex than RSA. We can check the following table for the comparison between ECDSA and RSA signatures.

Table 2. Comparison between ECDSA and RSA
Elliptic Curve Digital Signature AlgorithmRivest-Shamir-Adleman
ECDSA keys require less bandwidth. RSA keys require more bandwidth.
For any given level of security, ECDSA signatures are smaller. For the same level of security RSA signatures are bigger
It has been seen that ECDSA verification is little slower. RSA verification is little faster than ECDSA.

Creating ECDSA certificates on AIX and setting up IKEv2

Software requirements

To create an ECDSA certificate in IBM® AIX®, we need OpenSSL 1.01.502 and above to be installed in AIX. Also rename or remove RSA certificates (those are ikekey.crl, ikekey.kdb, ikekey.rdb, and ikekey.sth).

Creating the certificates

You need to perform the following steps to create the certificates.

Step 1: After the installation of OpenSSL, we need to change the openssl.cnf file from the /var/ssl path.

Change the following:

dir = /demoCA (it is the place where we will keep everything regarding certificates)

private_key = $dir/private/ec_cakey.pem

And, certificate = $dir/ec_ca.pem

Now, follow the given steps:

mkdir /demoCA /demoCA/certs /demoCA/crl /demoCA/private

We are using the /demoCA path as it is mentioned as a default in the openssl.cnf file. Users can change it to something else before creating the certificates and they should follow the corresponding path.

cd /demoCA
touch index.txt
echo "01" > serial

Step 2: Create the certificates.

  1. Generate an elliptic curve (EC) key for a new certificate authority (CA) certificate:

    To generate a ECDSA private key, we need to decide which elliptical curve to use. To know the list of the curves, we can use the openssl ecparam -list_curves command. Here in this example we are using the prime256v1 elliptical curve.

    openssl ecparam -genkey -text -name prime256v1 -out ec_cakey.pem
  2. Generate a certificate request for CA:

    Create a self-signed CA certificate containing the public key, and signed with the private key:

    openssl req -new -x509 -days 3650 -key ec_cakey.pem -sha256 -out ec_ca.pem

    This CA certificate can be used for both the ends, that is, for the initiator and the responder as we are using a single CA here.

    Generate a private key using the prime256v1 elliptical curve for end client certificates :

    openssl ecparam -genkey -text -name prime256v1 -out ec_aixtcp01-key.pem

    Follow the same for the other end client certificate.

  3. Generate a certificate request for the end client certificate using the following command.
    openssl req -new -sha256 -out ec_aixtcp01-req.pem -key ec_aixtcp01-key.pem -days 365

    Here, it asks for the email ID along with some other parameters. To work with IPSec , you should not give any email ID in it. Also, make sure that both of the end user certificates have different common names.

  4. Sign the certificate request using the root CA :
    openssl ca -out ec_aixtcp01.pem -days 365 -infiles ec_aixtcp01-req.pem
  5. Package it into a pkcs12 format:
    openssl pkcs12 -export -in ec_aixtcp01.pem -inkey ec_aixtcp01-key.pem -name "aixtcp01" -out

    Follow the same for the other end client certificate (say ec_aixtcp04.p12 for aixtcp04 or responder system).

  6. Copy both the files ec_ca.pem and ec_aixtcp01.p12 on the initiator (aixtcp01), and ec_ca.pem and ec_aixtcp04.p12 on the responder (aixtcp04) systems.

Step 3: Configure IPSec IKE in AIX, so that it can use ECDSA for IKE authentication.

Modify the /etc/ipsec/inet/DB/ec_secrets.conf file, uncomment ee and ca, and enter the correct path of ec_aixtcp01.p12 and ec_ca.pem in aixtcp01 (initiator), and ec_aixtcp04.p12 and ec_ca.pem in aixtcp04 (responder). The file now looks as shown in Figure 5.

Figure 5. ec_secrets.conf file

Step 4: Update the IKE DB to use ECDSA certificates and also to use IKEv2 instead of IKEv1.

It should look like the following:



 <ASN1_DN Value="/c=IN/st=Karnataka/o=IBM/ou=STG/cn=aixtcp04">
  <IPV4_Address Value=""/>
 <ASN1_DN Value="/c=IN/st=Karnataka/o=IBM/ou=STG/cn=aixtcp01"> 
  <IPV4_Address Value=""/>

And, in the IPSecProtection section:

<IPSecLocalIdentity Port="0" EndPort="65535" Protocol="0">
 <IPV4_Address_Range To_IPAddr="" From_IPAddr=""/>
<IPSecRemoteIdentity Port="0" EndPort="65535" Protocol="0">
 <IPV4_Address_Range To_IPAddr="" From_IPAddr=""/>

Step 5: Put the XML in IKE DB using the ikedb -p command. If the file is not maintaining a proper order, it will display an error.

Step 6: Run the smitty ipsec4 command to create an IPSec device and filter rules for ipv4.

Step 7: To capture log , you can modify /etc/isakmp.conf to set the log level to information and modify the syslog.conf file with a *.debug parameter to capture all the isakmp logs.

Run the startsrc -g ike command on both sides to start the tmd, cpsd, iked, isakmpd, and ikev2d daemons.

As IKE_Version="2" in IKE DB, ikev2d will be working for the current proposal.

Step 8: Now in the initiator system, run the following command:

ike cmd=activate

This activates both phase I and phase II tunnels.

Step 9: To check the tunnels, run ike cmd=list and ike cmd=list verbose.

Figure 6. ike cmd=list and ike cmd=list verbose


  • Check the ASN1_DN value on both sides to make sure that both sides are using proper certificates.
  • Check the /etc/ipsec/inet/DB/ec_secrets.conf file to verify the ee and ca paths and the password value.


Digital certificates are used to solve the problem of authenticity of the public key that it is not modified by the attacker. ECDSA provides the same security level with the corresponding RSA signature, but with much smaller size of certificate and public key. In this article, we have shown how we can use ECDSA for public key infrastructure for authentication in AIX IPSec IKEv2. Also, we have discussed how IKEv2 can be set up and how it works between two AIX systems.



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks

Zone=AIX and UNIX
ArticleTitle=Exploring IKEv2 with ECDSA certificate in IBM AIX