Troubleshooting HP-UX IPSec

IPSec Operation

On an end system (the local system is the source for the outbound packet), the Policy Manager sequentially searches the host IPSec policies in priority order for the first policy with an IP packet filter that matches the packet. If no match is found, HP-UX IPSec uses the default host IPSec policy.

On a gateway system (the local system is forwarding the outbound packet), the Policy Manager sequentially searches the gateway IPSec policies in priority order. If no match is found, HP-UX IPSec uses the default gateway IPSec policy.

If the transform (action) specified in the matching IPSec policy is to encrypt or authenticate the IP packets using AH or ESP, IPSec SAs may already exist for the policy. The new packet can use the existing IPSec/QM SAs if the IP addresses, ports and protocols match. The new packet can also use the existing IPSec/QM SAs if both IP addresses match and host-based keying is enabled (the Exclusive bit is not set for the policy). Otherwise, new IPSec/QM SAs are established.

3.Establish an ISAKMP/MM SA

Before establishing an IPSec/QM SA, the IKE daemon must establish an ISAKMP/MM SA with the remote system, if none exists. To establish an ISAKMP/MM SA, the Policy Manager daemon is queried for an IKE policy, based on the remote system’s IP address.

As part of the ISAKMP/MM SA negotiations, each system establishes a Diffie-Hellman value using the Oakley group specified in the IKE policy. Because the Diffie-Hellman algorithm is vulnerable to third-party attacks, each system must also authenticate the identity of the other system using the primary authentication method specified in the IKE policy (preshared keys or RSA signatures with security certificates), and verify the remote system’s ID type and ID value (HP-UX IPSec uses IP addresses by default). The two systems also negotiate the authentication and encryption algorithms for the ISAKMP/MM SA, and SA lifetime according to values specified in the IKE policies.

4.Establish IPSec/QM SAs

Once the ISAKMP/MM SA is established, the IKE daemon uses the secured channel to establish IPSec/QM SAs with its peers. Two IPSec/QM SAs are established: one for packets from the local system to the remote system, and one for packets from the remote system to the local system.

150

Chapter 5