Encapsulating Security Payload

The Encapsulating Security Payload (ESP) protocol provides data confidentiality, and also optionally provides data origin authentication, data integrity checking, and replay protection.

The difference between ESP and the Authentication Header (AH) protocol is that ESP provides encryption, while both protocols provide authentication, integrity checking, and replay protection. With ESP, both communicating systems use a shared key for encrypting and decrypting the data they exchange.

If you decide to use both encryption and authentication, then the responding system first authenticates the packet and then, if the first step succeeds, the system proceeds with decryption. This type of configuration reduces processing overhead, as well as reduces your vulnerability to denial-of-service attacks.

Two ways of using ESP

You can apply ESP in two ways: transport mode or tunnel mode. In transport mode, the ESP header follows the IP header of the original IP datagram. If the datagram already has an IPSec header, then the ESP header goes before it. The ESP trailer and the optional authentication data follow the payload.

Transport mode does not authenticate or encrypt the IP header, which might expose your addressing information to potential attackers while the datagram is in transit. Transport mode requires less processing overhead than tunnel mode, but does not provide as much security. In most cases, hosts use ESP in transport mode.

Tunnel mode creates a new IP header and uses it as the outermost IP header of the datagram, followed by the ESP header and then the original datagram (both the IP header and the original payload). The ESP trailer and the optional authentication data are appended to the payload. When you use both encryption and authentication, ESP completely protects the original datagram because it is now the payload data for the new ESP packet. ESP, however, does not protect the new IP header. Gateways must use ESP in tunnel mode.

What algorithms does ESP use to protect my information?

ESP uses a symmetric key that both communicating parties use to encrypt and decrypt the data they exchange. The sender and the receiver must agree on the key before secure communication takes place between them. VPN uses Data Encryption Standard (DES), triple-DES (3DES), Advanced Encryption Standard (AES), or AES-CBC and AES-CTR for encryption.

If you choose the AES algorithm for encryption then you might want to enable Extended Sequence Number (ESN). ESN allows you to transmit large volumes of data at a high speed. The VPN connection uses a 64-bit sequence numbers instead of 32-bit numbers over IPSec. Using 64-bit sequence numbers allows more time before re-keying, which prevents sequence number exhaustion and minimizes the use of system resources.

The Internet Engineering Task Force (IETF) formally defines the algorithms in the following Request for Comments (RFC):

  • DES in Request for Comment (RFC) 1829, The ESP DES-CBC Transform
  • 3DES in RFC 1851, The ESP Triple DES Transform
  • AES-CBC in RFC 3602, The AES-CBC Cipher Algorithm and Its Use with IPsec
  • AES-CTR in RFC 3686, Using Advanced Encryption Standard (AES) Counter Mode with IPSec Encapsulating Security Payload (ESP)

You can view these and other RFCs on the Internet at the following Web address: http://www.rfc-editor.org.

ESP uses HMAC-MD5, HMAC-SHA, HMAC-SHA-256, HMAC-SHA-384, HMAC-SHA-512, and AES-XCBC-MAC algorithms to provide authentication functions. Each of the algorithms take variable-length input data and a secret key to produce fixed-length output data (called a hash or MAC value). If the hashes of two messages match, then it is likely that the messages are the same.

The Internet Engineering Task Force (IETF) formally defines the algorithms in the following Request for Comments (RFC):

  • HMAC-MD5 in RFC 2085, HMAC-MD5 IP Authentication with Replay Prevention
  • HMAC-SHA in RFC 2404, The Use of HMAC-SHA-1-96 within ESP and AH
  • HMAC-SHA_256, HMAC-SHA-384, and HMAC-SHA-512 in RFC 4868, Using HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512 with IPsec
  • AES-XCBC-MAC in RFC 3566, The AES-XCBC-MAC-96 Algorithm and Its Use With IPsec

You can view these RFCs on the Internet at the following Web address: http://www.rfc-editor.org.

ESP uses AES-CCM and AES-GCM to provide encryption and authentication. An authentication algorithm cannot be selected if one of these "combined" algorithms is chosen.

  • AES-CCM in RFC 4309, Using Advanced Encryption Standard (AES) CCM mode with IPSec Encapsulating Security Payload (ESP)
  • AES-GCM in RFC 4106, The Use of Galios/Counter Mode (GCM) in IPSec Encapsulating Security Payload (ESP)

ESP uses AES-GMAC (Galios Message Authentication Code) to provide authentication, but not encryption.

  • AES-GMAC in RFC 4543, The Use of Galios Message Authentication Code (GMAC) in IPSec ESP and AH.