Security protocols and encapsulation modes

IPsec comes with two security protocols, AH and ESP. They define how to encapsulate IP packets and the security services that they can provide.

• AH (protocol 51) defines the encapsulation of the AH header in an IP packet, as shown in Figure 92. AH can provide data origin authentication, data integrity, and anti-replay services to prevent data tampering, but it cannot prevent eavesdropping. Therefore, it is suitable for transmitting non-confidential data. AH supports authentication algorithms HMAC-MD5 and HMAC-SHA1.

• ESP (protocol 50) defines the encapsulation of the ESP header and trailer in an IP packet, as shown in Figure 92. ESP can provide data encryption, data origin authentication, data integrity, and anti-replay services. Unlike AH, ESP can guarantee data confidentiality because it can encrypt the data before encapsulating the data to IP packets. ESP supports encryption algorithms such as DES, 3DES, and AES, and authentication algorithms HMAC-MD5 and HMAC-SHA1.

Both AH and ESP provide authentication services, but the authentication service provided by AH is stronger. In practice, you can choose either or both security protocols. When both AH and ESP are used, an IP packet is encapsulated first by ESP and then by AH.

IPsec supports the following encapsulation modes:

• Transport mode—The security protocols protect the upper layer data of an IP packet. Only the transport layer data is used to calculate the security protocol headers. The calculated security protocol headers and the encrypted data (only for ESP encapsulation) are placed after the original IP header. You can use the transport mode when end-to-end security protection is required (the secured transmission start and end points are the actual start and end points of the data). The transport mode is typically used for protecting host-to-host communications, as shown in Figure 90.

  Figure 89: IPsec protection in transport mode

• Tunnel mode—The security protocols protect the entire IP packet. The entire IP packet is used to calculate the security protocol headers. The calculated security protocol headers and the encrypted data (only for ESP encapsulation) are encapsulated in a new IP packet. In this mode, the encapsulated packet has two IP headers. The inner IP header is the original IP header. The outer IP header is added by the network device that provides the IPsec service. You must use the tunnel mode when the secured transmission start and end points are not the actual start and end points of the data packets (for example, when two gateways provide IPsec but the data start and end points are two hosts behind the gateways). The tunnel mode is typically used for protecting gateway-to-gateway communications, as shown in Figure 91.

  Figure 90: IPsec protection in tunnel mode

Figure 92 shows how the security protocols encapsulate an IP packet in different encapsulation modes.