NAT is incompatible with the AH protocol in both Transport and Tunnel mode. An IPSec VPN using the AH protocol digitally signs the outbound packet, both data payload and headers, with a hash value appended to the packet. When using AH protocol, packet contents (the data payload) are not encrypted.
A NAT device in between the IPSec endpoints will rewrite either the source or destination address with one of its own choosing. The VPN device at the receiving end will verify the integrity of the incoming packet by computing its own hash value, and complain that the hash value appended to the received packet doesn't match. The VPN device at the receiving end doesn't know about the NAT in the middle, so it assumes that the data has been maliciously altered.
IPSec using ESP in Tunnel mode encapsulates the entire original packet (including headers) in a new IP packet. The new IP packet's source address is the outbound address of the sending VPN gateway, and its destination address is the inbound address of the VPN device at the receiving end. When using ESP protocol with authentication, the packet contents (in this case, the entire original packet) are encrypted. The encrypted contents, but not the new headers, are signed with a hash value appended to the packet.
Tunnel mode ESP with authentication is compatible with NAT because integrity checks are performed over the combination of the "original header plus original payload," which is unchanged by a NAT device.
Transport mode ESP with authentication is not compatible with NAT.
Table 86 VPN and NAT
SECURITY PROTOCOL | MODE | NAT |
|
|
|
AH | Transport | N |
|
|
|
AH | Tunnel | N |
|
|
|
ESP | Transport | N |
|
|
|
ESP | Tunnel | Y |
|
|
|
Chapter 15 Introduction to IPSec | 237 |