Troubleshooting HP-UX IPSec

IPSec Operation

•Clear Text Packet

If the inbound packet has no AH or ESP (it is a normal IP packet in clear text), HP-UX IPSec must still determine whether the packet should be dropped or passed in clear text. HP-UX IPSec checks the kernel policy engine cache for an existing decision on the action to take (drop or pass in clear text) for the packet based on the IP addresses, protocol, and port numbers. If the action is to apply an AH or ESP transform, HP-UX IPSec sends an audit message to the audit daemon. This is because the remote system should have established IPSec/QM SAs before sending the packet.

If no cache entry exists, HP-UX IPSec queries the policy manager daemon for the appropriate action according to the host IPSec policy with the filter that best matches the packet (or the default policy, if no filters match). Again, if the action is to apply an AH or ESP transform, HP-UX IPSec discards the packet and sends an audit message to the audit daemon.

Establishing Tunnel Security Associations

If HP-UX IPSec is processing an outbound packet and the selected host or gateway IPSec policy specifies a tunnel IPSec policy, HP-UX IPSec checks if it has an existing tunnel SA with the tunnel endpoint. If not, it must establish a tunnel SA before it establishes the end-to-end (transport) SA. The procedure for establishing a tunnel SA is similar to establishing a transport SA (HP-UX IPSec uses/establishes an ISAKMP/MM SA to establish the IPSec/QM SA), except the IKE entities also include proxy address information during IPSec/QM SA negotiation. The proxy address information identifies the end-to-end entities and allows a tunnel endpoint to determine the identity of the end system or subnet for which the other tunnel endpoint is establishing the tunnel.

Processing Inbound Tunnel Packets

If HP-UX IPSec is processing an inbound packet, it searches the kernel SA database for inbound packets for an entry with the same SPI and source IP address. If one exists, it uses the information in the SA to decrypt or authenticate the packet. If this is a tunnel SA, HP-UX IPSec decapsulates the packet (removes the outer IP header) and processes the IP header for the inner packet. If the destination address in the inner packet is a local address, HP-UX IPSec searches its host IPSec policies to determine the next action. If it is not a local address, HP-UX IPSec searches its gateway IPSec policies to determine the next action. If the

Chapter 5

153