HP-UX IPSec version A.02.00 Administrator’s Guide
Manufacturing Part Number J4256-90009 June
Legal Notices
Contents
Configuring HP-UX IPSec
Contents
Using Certificates with HP-UX IPSec
Troubleshooting HP-UX IPSec
Viii
HP-UX IPSec and IPFilter
HP-UX IPSec and HP-UX Mobile IPv6
HP-UX IPSec and MC/ServiceGuard
Xii
HP-UX IPSec and Linux
Migrating from Previous Versions of HP-UX IPSec
Glossary
Xvi
Tables
Xviii
Figures
Figure C-2. Example 1 telnet BA
Intended Audience
New and Changed Documentation in This Edition
Xxii
What’s in This Document
Publishing History
HP-UX IPSec and HP-UX Mobile IPv6 Use this chapter to learn
Related Documents
Typographical Conventions
HP Encourages Your Comments
OpenSSL Copyright Notice
Xxvi
Xxvii
Xxviii
HP-UX IPSec Overview
HP-UX IPSec Overview
Introduction
Introduction
Authentication Header AH
Transport and Tunnel Modes
Symmetric Key Authentication
Transport Mode
Host a
AH in Transport Mode
Tunnel Mode
AH in Tunnel Mode
ESP Encryption
Encapsulating Security Payload ESP
Symmetric Key Cryptosystem
ESP header can be used in transport mode or tunnel mode
ESP in Tunnel Mode
ESP Encryption in Transport Mode
IP data or payload e.g., TCP or UDP packet
ESP with Authentication and Encryption
Authenticated ESP
Nested ESP in AH
IPv6
Security Associations SAs and IKE Phases
Internet Key Exchange IKE
Generating Shared Keys Diffie-Hellman
SA Establishment
IKE Primary Authentication
10 Diffie-Hellman Key Generation
IKE Automatic Re-keying
Re-using Negotiations
IKE Preshared Key Authentication
Digital Signatures
Manual Keys
Host-to-Host Topology
HP-UX IPSec Topologies
Host-to-Gateway Topology
Host-to-Host Tunnel Topology
13 Host-to-Host Tunnel Topology
Gateway-to-Gateway Topology
14 IPSec Gateway-to-Gateway Topology
HP-UX IPSec Configuration and Management Features
HP-UX IPSec Configuration and Management Features
HP-UX IPSec Configuration and Management Features Chapter
Installing HP-UX IPSec
Installing HP-UX IPSec
HP-UX IPSec Product Requirements
Security Certificate Configuration Utility Requirements
Disk Requirements
Chapter
Loading the HP-UX IPSec Software
Do not run the HP-UX IPSec product when the system is booted
Re-establishing the HP-UX IPSec Password
Setting the HP-UX IPSec Password
Ipsecadmin -newpasswd
Completing Post-Installation Migration Requirements
Configuring HP-UX IPSec
Configuring HP-UX IPSec
Bypass List
Maximizing Security
Strong End System Model
Ndd -set /dev/ip ipstrongesmodel
Argument Delimiters
General Syntax Information
Line Continuation Character \
Batch File Processing
Batch File Syntax
Ipsecconfig delete
Profile File
Profile File Structure
Using a Profile File with a Batch File
Creating a Customized Profile File
Dynamic Configuration Updates
Dynamic Deletions
Configuration Overview
Start-up options
Configuration Overview
Default Host IPSec Policy
Configuring Host IPSec Policies
Policy Order and Selection
Automatic Priority Increment
Ipsecconfig add host hostpolicyname
Action PASSDISCARDtransformlist -flags flags
Hostpolicyname
Ipaddr/prefix/portnumberservicename
Source and -destination
Ipsecconfig Service Names
Service Port Protocol Name
Ipsecconfig Service Names
Protocolprotocolid
Priorityprioritynumber
Default ALL
Tunneltunnelpolicyname
Action
Transformname/lifetimeseconds/lifetimekbytes
Transformname
Ipsecconfig Transforms
Transform Name Description
Ipsecconfig Transforms
ESP3DES
Flags flags
Ipsecconfig add host Flags
Flag Description
Host IPSec Policy Configuration Examples
Configuring Host IPSec Policies
Configuring Tunnel IPSec Policies
Ipsecconfig add tunnel tunnelpolicyname
Tunnelpolicyname
Default None
Tsource and -tdestination tunneladdress
Ipaddr/prefix/portnumberservicename
Subnet address filter
TCP UDP Icmp ICMPV6 Igmp
Actiontransformlist
Lifetimeseconds
Tunnel IPSec Policy Configuration Example
Lifetimekbytes
Configuring Tunnel IPSec Policies
Configuring IKE Policies
Add ike ikepolicyname
Lifelifetimeseconds -maxqmmqmaxquickmodes
Ikepolicyname
Remoteipaddr/prefix
Acceptable Values
Authenticationauthenticationtype
Group
Lifelifetimeseconds
Hash MD5SHA1
Maxqmmaxquickmodes
Ipsecconfig add IKE Command Examples
Default
Configuring IKE ID Information with Preshared Keys
Configuring Preshared Keys Using Authentication Records
Remote Multi-homed Systems
Remoteipaddr/prefix -presharedpresharedkey
Ipsecconfig add auth authname
Add auth authname
Authname
Ipaddr/prefix
Unique preshared key
Authentication Record Configuration Examples
However, HP strongly recommends that you configure an
Presharedkey
Configuring Preshared Keys Using Authentication Records
Configuring Certificates
Logical Interfaces
Configuring the Bypass List Local IPv4 Addresses
Example
Bypass List Example
Ipsecconfig add bypass ipaddress
Maximizing Security
Node1 Node2
Add bypass ipaddress
Bypass Configuration Example
Ipaddress
Ipsecconfig batch batchfilename -nocommit
Verify Batch File Syntax
Ipsecconfig batch batchfilename
Ipsecconfig show all
Ipsecadmin -status
Ipsecreport -cache
Ipsecreport -all
108
Ipsecconfig add startup -autoboot on
Configuring HP-UX IPSec to Start Automatically
Add startup -autoboot on
110
Baltimore Configuration Files
VeriSign Configuration Files
112
Using Certificates with HP-UX
114
Security Certificates and Public Key Cryptography
Overview
Public Key Distribution
Digital Signatures
IKE Public Key Distribution
Requirements
Using VeriSign Certificates
Overview
VeriSign PKI Data Flow
VeriSign Certificate Tasks
Step
Verifying Prerequisites
Ipsecmgr
Configuring Web Proxy Server Parameters
Export DISPLAY=displaydevice0.0
Registering the Administrator
Requesting and Receiving Certificates
124
Chapter 125
Using Baltimore Certificates
Baltimore Certificate Tasks
Chapter 127
Requesting the Baltimore Certificate
Configuring the Baltimore Certificate
130
Chapter 131
132
Chapter 133
Configuring Authentication Records with IKE IDs
Chapter 135
VeriSign SubjectAlternativeName
Determining the IPv4 Address in the SubjectAlternativeName
Syntax
Add auth authname -remoteipaddr/prefix
Ltypelocalidtype
Lvalue localid
Rtyperemoteidtype
Ridremoteid
CN=commonName,O=organization,C=country,OU=organizationUnit
Examples
Add auth Black -remote 10.10.10.10 -ltype IPV4 \ -lid
VeriSign
Retrieving the Certificate Revocation List CRL
Baltimore
Manually Retrieving a CRL for VeriSign or Baltimore
144
Troubleshooting HP-UX IPSec
146
IPSec Operation
Authenticate Each Peer’s Identity
Authenticate Identities
Establishing Security Associations SAs
Establish ISAKMP/MM SA
Establish IPSec/QM SAs
Outbound Data Query the Kernel Policy Engine
Internal Processing
Outbound Processing
Query the Policy Manager Daemon
Establish an ISAKMP/MM SA
Add IPSec/QM SAs to the Kernel SA Database
Inbound Data AH or ESP Packet
152
Establishing Tunnel Security Associations
Clear Text Packet
Processing Inbound Tunnel Packets
154
Troubleshooting Utilities Overview
Getting SA Information
Getting General Information
Getting Policy Information
Configured
Ipsecreport -host configured
Ipsecconfig show gateway
Ipsecconfig show tunnel
Viewing and Configuring Audit Information
Getting Interface Information
Enabling and Disabling Tracing
Checking Status
Troubleshooting Procedures
Ipsecreport -all -file filename
Chapter 161
Isolating HP-UX IPSec Problems from Upper-layer
Ipsecadmin -traceon tcp udp igmp all
Using ipsecpolicy
Checking Policy Configuration
Examining the Policy Cache and Policy Entries
Configuring HP-UX IPSec Auditing
Audit Level
Ipsecadmin -al auditlevel -au auditdirectory
Ipsecadmin -maxsize maxauditfilesize
Dynamically Setting Audit Parameters
Audit Files and Directory
Viewing Audit Files
Configuring Startup Audit Parameters
Ipsecconfig add startup -autoboot Onoff
Auditlvlauditlevel -auditdirauditdirectory
Recorded by specified entities
Where entityname is one of the following names
Filtering Audit File Output by Entity
Ipsecreport -audit auditfile -entity entityname
Reporting Problems
Output from ipsecadmin -status Output from ipsecreport -all
Chapter 169
Problem
Troubleshooting Scenarios
HP-UX IPSec Incorrectly Passes Packets
Symptoms
Solution
HP-UX IPSec Attempts to Encrypt/Authenticate and Fails
Additional Information
Ipsecreport -mad Ipsecreport -audit file
Ipsecreport -audit /var/adm/ipsec/auditdateinfo.log
ISAKMP/MM SA Negotiation Fails Main Mode
Processing failed, MM negotiation timeout
Chapter 175
Isakmp Primary Authentication with Preshared Key Fails
Isakmp Primary Authentication Fails with Certificates
ISAKMP/MM SA Negotiation Succeeded, IPSec/QM SA
Details
Manual Keys Fail
Invalid Sadbadd
Nettl -log e d -e streams
Streams Logging Messages and Additional Audit File Entries
Nettl -ss
Netfmt /var/adm/nettl.LOG000 mylogoutput
HP-UX Will Not Start ipsecadmin -startFails
Ipsecadmin -auditlvl warning
182
Corrupt or Missing Configuration Database
Ipsecmigrate -s oldconfigfile -d newconfigfile
Using the Skeleton Database File
Autoboot is Not Working Properly
Administrator Cannot Get a Local VeriSign Certificate
186
Security Policy Database Limit Exceeded Kernel
188
HP-UX IPSec and IPFilter
190
IPFilter and IPSec
IPFilter and IPSec Basics
IPFilter Scenario One
192
IPSec UDP Negotiation
IPFilter Scenario Two
194
When Traffic Appears to be Blocked
Scenario Three
Packet with Encrypted TCP Data
Allowing Protocol 50 and Protocol 51 Traffic
Packet with IPSec-Encrypted TCP Data
Protocol 51 traffic, then IPSec traffic will not get through
Scenario Four
IPSec Gateways
HP-UX IPSec and HP-UX Mobile
200
Correspondent Nodes
Mobile Node and Home Address
Care-of Address
Home Agent
Mobile IPv6 Basic Operation Correspondent Node to Mobile
Home Agents and Basic Operation
Node
Mobile IPv6 Basic Operation Mobile Node to Correspondent
Route Optimization
Mobile IPv6 Route Optimization
Prefix Discovery Messages
Securing Mobile IPv6 with HP-UX IPSec
Acknowledgement messages
Binding Messages Between the Home Agent and Mobile Node
Chapter 205
Payload Packets Routed Through the Home Agent
Chapter 207
Understanding Gateway IPSec Policies
Gateway IPSec Policies
Using the HP-UX Strong Random Number Generator
Using Manual Keys
Configuration Procedure
Troubleshooting Manual Key Problems
210
Syntax
Sourcehomeagentaddr
Inand -outmanualkeysaspecification
Actiontransformname
Chapter 213
2B, 2C Home Agent Mobile Node
Mobile IPv6 Home Test Init and Home Test Packets
Gateway IPSec Policy for Home Agent
Return Routability Messages Configuring
Correspondent Node Segments
216
Tunnel rrtunnelname -action Forward -flags MIPV6
Tunnelrrtunnelname
218
Chapter 219
220
Chapter 221
222
Protocol ALL -priority prioritynumber
Action Forward -flags MIPV6
224
Tunnelpayloadtunnelname
Ipsecconfig add tunnel payloadtunnelname
Return Routability Messages
Mobile IPv6 Configuration Example
Binding Messages
3ffe83fffef71111
Gateway IPSec Policy for Home Agent Mobile Node Segments
Optional Prefix Discovery Messages
Return Routability Tunnel IPSec Policy
Add gateway mn2222payloadtocn \
Optional Payload Messages Routed Through the Home Agent
Payload Gateway IPSec Policies
Protocol ALL -pri 300 -action Forward -flags MIPV6
Payload Tunnel IPSec Policy
Batch File Template
232
Chapter 233
234
HP-UX IPSec
236
MC/ServiceGuard Cluster
Using HP-UX IPSec with MC/ServiceGuard
Package Clients Not Using HP-UX IPSec A.01.07 or Later
Chapter 239
MC/ServiceGuard Heartbeat Requirement Recommendation
Configuration Steps
242
Configuring a Common HP-UX IPSec Password
Configuring HP-UX Host IPSec Policies for MC/ServiceGuard
Determining MC/ServiceGuard Cluster Information
Configuring Host IPSec Policies for Package Addresses
Private Dedicated Heartbeat Networks
1238
10.0.0.0/8
Cluster Node IPSec Policies for Quorum Server
Address or Server Wildcard
Server Address Address or Wildcard
Cluster Node IPSec Policies for Remote Command Execution
Quorum Server IPSec Policies
Source IP Destination Protocol Address IP Address Port
Address or Command Wildcard Client address
514
Configuring Host IPSec Policies for ServiceGuard Manager
Command Address Client address Or wildcard
ServiceGuard Manager Host IPSec Policies
Cluster Node Host IPSec Policies for ServiceGuard Manager
Source IP Destination Protoco Address IP Address Port
Cluster Node Host IPSec Policies for COM
COM System Host IPSec Policies
Port Protocols Service
Summary MC/ServiceGuard Port Numbers Protocols
MC/ServiceGuard Port Numbers and Protocols
5303
Chapter 255
256
Cluster IKE policies
Configuring HP-UX IPSec IKE policies
Cluster Client IKE policies
Preshared Key Configuration on Cluster Nodes
Configuring Authentication Records for Preshared Keys
Preshared Key Configuration on Client Nodes
Preshared Keys Configuration on Client2
Preshared Keys Configuration on Cluster Nodes
Preshared Keys Configuration on Client1
Remote IP Address Key
260
Configuring Authentication Records for Certificates
Authentication Records and IKE ID Information
Cluster Clients
Chapter 263
IKE ID Configuration on Cluster Nodes
IKE ID Configuration on Client1 and Client2
Verifying and Testing the HP-UX IPSec Configuration
Ipsecpolicy -sa 15.1.1.1 -da
Configuring HP-UX IPSec Start-up Options
Distributing HP-UX IPSec Configuration Files
268
Package Configuration
Configuring MC/ServiceGuard
Cluster Configuration
Package Control Script
Monitor Script Polling Interval
Starting HP-UX IPSec MC/ServiceGuard
Adding a Node to a Running Cluster
272
HP-UX IPSec and Linux
274
Chapter 275
Configuration Example
Product Specifications
Appendix a
RFC 3776 Mandatory Support
IPSec RFCs
RFC Number RFC Title
280
Product Restrictions
Isakmp Limitations
IPv4 Icmp Messages
IPv6 Icmp Messages
Comparative Key Lengths
Authentication Algorithms
HP-UX IPSec Transforms
Algorithm Key Length
Encryption Algorithms
ESP-DES
Transform Lifetime Negotiation
Migrating from Previous Versions
Appendix B
Migrating from Versions Prior to A.01.03
Pre-Installation Migration Instructions
MD5 Version Compatibility
Ipsecreport -auditauditfilename -fileoutputfilename
Not Re-using Configuration Files
Usr/sbin/ipsecmigrate -s configfile -d newconfigfile
Post-Installation Migration Instructions
Configuration File
Ipsecadmin -start
292
HP-UX IPSec Configuration Examples
Appendix C
Example 1 telnet Between Two Systems
Apple Configuration
Figure C-1 Example 1 telnet AB
Authentication Record with Preshared Key
Banana Configuration
IKE Policy
298
Example 2 Authenticated ESP with Exceptions
Figure C-3 Example 2 Network IPSec Policy with Exceptions
Carrot Configuration
Ipsecconfig Batch File Entries
Authentication Record
Host IPSec Policy
Blue Configuration
Example 3 Host to Gateway
Priority 100 -action Pass -tunnel torouter
Tunnel IPSec Policy
Add auth torouter -rem 16.6.6.6 -psk Hello
Cat Configuration
Dog Configuration
Example 4 Manual Keys
Glossary
Asymmetric keys, public/private keys
Diffie-Hellman
Encapsulating Security Payload ESP
Glossary 307
Preshared Key
Numerics
309
310
311
312
313
314