Manufacturing Part Number J4256-90009 June
HP-UX IPSec version A.02.00 Administrator’s Guide
Legal Notices
Contents
Configuring HP-UX IPSec
Contents
Using Certificates with HP-UX IPSec
Troubleshooting HP-UX IPSec
Viii
HP-UX IPSec and IPFilter
HP-UX IPSec and HP-UX Mobile IPv6
HP-UX IPSec and MC/ServiceGuard
Xii
HP-UX IPSec and Linux
Migrating from Previous Versions of HP-UX IPSec
Glossary
Xvi
Tables
Xviii
Figures
Figure C-2. Example 1 telnet BA
New and Changed Documentation in This Edition
Intended Audience
Xxii
Publishing History
What’s in This Document
HP-UX IPSec and HP-UX Mobile IPv6 Use this chapter to learn
Typographical Conventions
Related Documents
HP Encourages Your Comments
Xxvi
OpenSSL Copyright Notice
Xxvii
Xxviii
HP-UX IPSec Overview
HP-UX IPSec Overview
Introduction
Introduction
Authentication Header AH
Host a
Symmetric Key Authentication
Transport Mode
Transport and Tunnel Modes
AH in Transport Mode
AH in Tunnel Mode
Tunnel Mode
Encapsulating Security Payload ESP
ESP Encryption
Symmetric Key Cryptosystem
ESP header can be used in transport mode or tunnel mode
ESP Encryption in Transport Mode
ESP in Tunnel Mode
IP data or payload e.g., TCP or UDP packet
Authenticated ESP
ESP with Authentication and Encryption
IPv6
Nested ESP in AH
Internet Key Exchange IKE
Security Associations SAs and IKE Phases
SA Establishment
Generating Shared Keys Diffie-Hellman
10 Diffie-Hellman Key Generation
IKE Primary Authentication
Digital Signatures
Re-using Negotiations
IKE Preshared Key Authentication
IKE Automatic Re-keying
Manual Keys
HP-UX IPSec Topologies
Host-to-Host Topology
Host-to-Gateway Topology
13 Host-to-Host Tunnel Topology
Host-to-Host Tunnel Topology
14 IPSec Gateway-to-Gateway Topology
Gateway-to-Gateway Topology
HP-UX IPSec Configuration and Management Features
HP-UX IPSec Configuration and Management Features
HP-UX IPSec Configuration and Management Features Chapter
Installing HP-UX IPSec
Installing HP-UX IPSec
Security Certificate Configuration Utility Requirements
HP-UX IPSec Product Requirements
Disk Requirements
Chapter
Loading the HP-UX IPSec Software
Do not run the HP-UX IPSec product when the system is booted
Setting the HP-UX IPSec Password
Re-establishing the HP-UX IPSec Password
Ipsecadmin -newpasswd
Completing Post-Installation Migration Requirements
Configuring HP-UX IPSec
Configuring HP-UX IPSec
Maximizing Security
Bypass List
Strong End System Model
Ndd -set /dev/ip ipstrongesmodel
General Syntax Information
Argument Delimiters
Line Continuation Character \
Batch File Syntax
Batch File Processing
Profile File
Ipsecconfig delete
Using a Profile File with a Batch File
Profile File Structure
Creating a Customized Profile File
Dynamic Deletions
Dynamic Configuration Updates
Configuration Overview
Start-up options
Configuration Overview
Configuring Host IPSec Policies
Default Host IPSec Policy
Policy Order and Selection
Ipsecconfig add host hostpolicyname
Automatic Priority Increment
Action PASSDISCARDtransformlist -flags flags
Ipaddr/prefix/portnumberservicename
Hostpolicyname
Source and -destination
Service Port Protocol Name
Ipsecconfig Service Names
Protocolprotocolid
Ipsecconfig Service Names
Default ALL
Priorityprioritynumber
Tunneltunnelpolicyname
Transformname/lifetimeseconds/lifetimekbytes
Action
Ipsecconfig Transforms
Transformname
Transform Name Description
ESP3DES
Ipsecconfig Transforms
Ipsecconfig add host Flags
Flags flags
Flag Description
Host IPSec Policy Configuration Examples
Configuring Host IPSec Policies
Ipsecconfig add tunnel tunnelpolicyname
Configuring Tunnel IPSec Policies
Default None
Tunnelpolicyname
Tsource and -tdestination tunneladdress
Ipaddr/prefix/portnumberservicename
Subnet address filter
TCP UDP Icmp ICMPV6 Igmp
Actiontransformlist
Tunnel IPSec Policy Configuration Example
Lifetimeseconds
Lifetimekbytes
Configuring Tunnel IPSec Policies
Configuring IKE Policies
Lifelifetimeseconds -maxqmmqmaxquickmodes
Add ike ikepolicyname
Ikepolicyname
Remoteipaddr/prefix
Authenticationauthenticationtype
Acceptable Values
Group
Hash MD5SHA1
Lifelifetimeseconds
Maxqmmaxquickmodes
Default
Ipsecconfig add IKE Command Examples
Configuring Preshared Keys Using Authentication Records
Configuring IKE ID Information with Preshared Keys
Remote Multi-homed Systems
Ipsecconfig add auth authname
Remoteipaddr/prefix -presharedpresharedkey
Add auth authname
Ipaddr/prefix
Authname
Presharedkey
Authentication Record Configuration Examples
However, HP strongly recommends that you configure an
Unique preshared key
Configuring Preshared Keys Using Authentication Records
Configuring Certificates
Configuring the Bypass List Local IPv4 Addresses
Logical Interfaces
Example
Node1 Node2
Ipsecconfig add bypass ipaddress
Maximizing Security
Bypass List Example
Bypass Configuration Example
Add bypass ipaddress
Ipaddress
Verify Batch File Syntax
Ipsecconfig batch batchfilename -nocommit
Ipsecconfig show all
Ipsecconfig batch batchfilename
Ipsecreport -cache
Ipsecadmin -status
Ipsecreport -all
108
Configuring HP-UX IPSec to Start Automatically
Ipsecconfig add startup -autoboot on
Add startup -autoboot on
110
VeriSign Configuration Files
Baltimore Configuration Files
112
Using Certificates with HP-UX
114
Overview
Security Certificates and Public Key Cryptography
Public Key Distribution
IKE Public Key Distribution
Digital Signatures
Requirements
Overview
Using VeriSign Certificates
VeriSign Certificate Tasks
VeriSign PKI Data Flow
Step
Verifying Prerequisites
Configuring Web Proxy Server Parameters
Ipsecmgr
Export DISPLAY=displaydevice0.0
Registering the Administrator
Requesting and Receiving Certificates
124
Chapter 125
Baltimore Certificate Tasks
Using Baltimore Certificates
Chapter 127
Requesting the Baltimore Certificate
Configuring the Baltimore Certificate
130
Chapter 131
132
Chapter 133
Configuring Authentication Records with IKE IDs
Chapter 135
Determining the IPv4 Address in the SubjectAlternativeName
VeriSign SubjectAlternativeName
Syntax
Add auth authname -remoteipaddr/prefix
Lvalue localid
Ltypelocalidtype
Ridremoteid
Rtyperemoteidtype
Examples
CN=commonName,O=organization,C=country,OU=organizationUnit
Add auth Black -remote 10.10.10.10 -ltype IPV4 \ -lid
Retrieving the Certificate Revocation List CRL
VeriSign
Baltimore
Manually Retrieving a CRL for VeriSign or Baltimore
144
Troubleshooting HP-UX IPSec
146
Establishing Security Associations SAs
Authenticate Each Peer’s Identity
Authenticate Identities
IPSec Operation
Establish IPSec/QM SAs
Establish ISAKMP/MM SA
Query the Policy Manager Daemon
Internal Processing
Outbound Processing
Outbound Data Query the Kernel Policy Engine
Establish an ISAKMP/MM SA
Inbound Data AH or ESP Packet
Add IPSec/QM SAs to the Kernel SA Database
152
Clear Text Packet
Establishing Tunnel Security Associations
Processing Inbound Tunnel Packets
154
Troubleshooting Utilities Overview
Getting General Information
Getting SA Information
Getting Policy Information
Ipsecconfig show tunnel
Ipsecreport -host configured
Ipsecconfig show gateway
Configured
Getting Interface Information
Viewing and Configuring Audit Information
Enabling and Disabling Tracing
Troubleshooting Procedures
Checking Status
Ipsecreport -all -file filename
Chapter 161
Ipsecadmin -traceon tcp udp igmp all
Isolating HP-UX IPSec Problems from Upper-layer
Checking Policy Configuration
Using ipsecpolicy
Examining the Policy Cache and Policy Entries
Audit Level
Configuring HP-UX IPSec Auditing
Audit Files and Directory
Ipsecadmin -maxsize maxauditfilesize
Dynamically Setting Audit Parameters
Ipsecadmin -al auditlevel -au auditdirectory
Auditlvlauditlevel -auditdirauditdirectory
Configuring Startup Audit Parameters
Ipsecconfig add startup -autoboot Onoff
Viewing Audit Files
Ipsecreport -audit auditfile -entity entityname
Where entityname is one of the following names
Filtering Audit File Output by Entity
Recorded by specified entities
Output from ipsecadmin -status Output from ipsecreport -all
Reporting Problems
Chapter 169
Troubleshooting Scenarios
Problem
HP-UX IPSec Incorrectly Passes Packets
Solution
Symptoms
HP-UX IPSec Attempts to Encrypt/Authenticate and Fails
Ipsecreport -mad Ipsecreport -audit file
Additional Information
Ipsecreport -audit /var/adm/ipsec/auditdateinfo.log
Processing failed, MM negotiation timeout
ISAKMP/MM SA Negotiation Fails Main Mode
Chapter 175
Isakmp Primary Authentication Fails with Certificates
Isakmp Primary Authentication with Preshared Key Fails
Details
ISAKMP/MM SA Negotiation Succeeded, IPSec/QM SA
Manual Keys Fail
Invalid Sadbadd
Netfmt /var/adm/nettl.LOG000 mylogoutput
Streams Logging Messages and Additional Audit File Entries
Nettl -ss
Nettl -log e d -e streams
Ipsecadmin -auditlvl warning
HP-UX Will Not Start ipsecadmin -startFails
182
Corrupt or Missing Configuration Database
Using the Skeleton Database File
Ipsecmigrate -s oldconfigfile -d newconfigfile
Administrator Cannot Get a Local VeriSign Certificate
Autoboot is Not Working Properly
186
Security Policy Database Limit Exceeded Kernel
188
HP-UX IPSec and IPFilter
190
IPFilter and IPSec Basics
IPFilter and IPSec
IPFilter Scenario One
192
IPFilter Scenario Two
IPSec UDP Negotiation
194
Scenario Three
When Traffic Appears to be Blocked
Allowing Protocol 50 and Protocol 51 Traffic
Packet with Encrypted TCP Data
Packet with IPSec-Encrypted TCP Data
Scenario Four
Protocol 51 traffic, then IPSec traffic will not get through
IPSec Gateways
HP-UX IPSec and HP-UX Mobile
200
Home Agent
Mobile Node and Home Address
Care-of Address
Correspondent Nodes
Home Agents and Basic Operation
Mobile IPv6 Basic Operation Correspondent Node to Mobile
Node
Route Optimization
Mobile IPv6 Basic Operation Mobile Node to Correspondent
Mobile IPv6 Route Optimization
Binding Messages Between the Home Agent and Mobile Node
Securing Mobile IPv6 with HP-UX IPSec
Acknowledgement messages
Prefix Discovery Messages
Chapter 205
Payload Packets Routed Through the Home Agent
Chapter 207
Gateway IPSec Policies
Understanding Gateway IPSec Policies
Troubleshooting Manual Key Problems
Using Manual Keys
Configuration Procedure
Using the HP-UX Strong Random Number Generator
210
Syntax
Inand -outmanualkeysaspecification
Sourcehomeagentaddr
Actiontransformname
Chapter 213
Mobile IPv6 Home Test Init and Home Test Packets
2B, 2C Home Agent Mobile Node
Return Routability Messages Configuring
Gateway IPSec Policy for Home Agent
Correspondent Node Segments
216
Tunnelrrtunnelname
Tunnel rrtunnelname -action Forward -flags MIPV6
218
Chapter 219
220
Chapter 221
222
Action Forward -flags MIPV6
Protocol ALL -priority prioritynumber
224
Tunnelpayloadtunnelname
Ipsecconfig add tunnel payloadtunnelname
3ffe83fffef71111
Mobile IPv6 Configuration Example
Binding Messages
Return Routability Messages
Optional Prefix Discovery Messages
Gateway IPSec Policy for Home Agent Mobile Node Segments
Return Routability Tunnel IPSec Policy
Protocol ALL -pri 300 -action Forward -flags MIPV6
Optional Payload Messages Routed Through the Home Agent
Payload Gateway IPSec Policies
Add gateway mn2222payloadtocn \
Payload Tunnel IPSec Policy
Batch File Template
232
Chapter 233
234
HP-UX IPSec
236
MC/ServiceGuard Cluster
Package Clients Not Using HP-UX IPSec A.01.07 or Later
Using HP-UX IPSec with MC/ServiceGuard
Chapter 239
MC/ServiceGuard Heartbeat Requirement Recommendation
Configuration Steps
242
Configuring a Common HP-UX IPSec Password
Configuring HP-UX Host IPSec Policies for MC/ServiceGuard
Configuring Host IPSec Policies for Package Addresses
Determining MC/ServiceGuard Cluster Information
Private Dedicated Heartbeat Networks
Address or Server Wildcard
10.0.0.0/8
Cluster Node IPSec Policies for Quorum Server
1238
Source IP Destination Protocol Address IP Address Port
Cluster Node IPSec Policies for Remote Command Execution
Quorum Server IPSec Policies
Server Address Address or Wildcard
Address or Command Wildcard Client address
514
Command Address Client address Or wildcard
Configuring Host IPSec Policies for ServiceGuard Manager
Cluster Node Host IPSec Policies for ServiceGuard Manager
ServiceGuard Manager Host IPSec Policies
Source IP Destination Protoco Address IP Address Port
COM System Host IPSec Policies
Cluster Node Host IPSec Policies for COM
5303
Summary MC/ServiceGuard Port Numbers Protocols
MC/ServiceGuard Port Numbers and Protocols
Port Protocols Service
Chapter 255
256
Configuring HP-UX IPSec IKE policies
Cluster IKE policies
Cluster Client IKE policies
Configuring Authentication Records for Preshared Keys
Preshared Key Configuration on Cluster Nodes
Preshared Key Configuration on Client Nodes
Remote IP Address Key
Preshared Keys Configuration on Cluster Nodes
Preshared Keys Configuration on Client1
Preshared Keys Configuration on Client2
260
Authentication Records and IKE ID Information
Configuring Authentication Records for Certificates
Cluster Clients
Chapter 263
IKE ID Configuration on Client1 and Client2
IKE ID Configuration on Cluster Nodes
Ipsecpolicy -sa 15.1.1.1 -da
Verifying and Testing the HP-UX IPSec Configuration
Configuring HP-UX IPSec Start-up Options
Distributing HP-UX IPSec Configuration Files
268
Package Control Script
Configuring MC/ServiceGuard
Cluster Configuration
Package Configuration
Monitor Script Polling Interval
Adding a Node to a Running Cluster
Starting HP-UX IPSec MC/ServiceGuard
272
HP-UX IPSec and Linux
274
Chapter 275
Configuration Example
Product Specifications
Appendix a
IPSec RFCs
RFC 3776 Mandatory Support
RFC Number RFC Title
280
Isakmp Limitations
Product Restrictions
IPv4 Icmp Messages
IPv6 Icmp Messages
Algorithm Key Length
Authentication Algorithms
HP-UX IPSec Transforms
Comparative Key Lengths
ESP-DES
Encryption Algorithms
Transform Lifetime Negotiation
Migrating from Previous Versions
Appendix B
Ipsecreport -auditauditfilename -fileoutputfilename
Pre-Installation Migration Instructions
MD5 Version Compatibility
Migrating from Versions Prior to A.01.03
Not Re-using Configuration Files
Ipsecadmin -start
Post-Installation Migration Instructions
Configuration File
Usr/sbin/ipsecmigrate -s configfile -d newconfigfile
292
HP-UX IPSec Configuration Examples
Appendix C
Example 1 telnet Between Two Systems
Figure C-1 Example 1 telnet AB
Apple Configuration
Banana Configuration
Authentication Record with Preshared Key
IKE Policy
298
Figure C-3 Example 2 Network IPSec Policy with Exceptions
Example 2 Authenticated ESP with Exceptions
Carrot Configuration
Authentication Record
Ipsecconfig Batch File Entries
Priority 100 -action Pass -tunnel torouter
Blue Configuration
Example 3 Host to Gateway
Host IPSec Policy
Add auth torouter -rem 16.6.6.6 -psk Hello
Tunnel IPSec Policy
Dog Configuration
Cat Configuration
Example 4 Manual Keys
Asymmetric keys, public/private keys
Glossary
Encapsulating Security Payload ESP
Diffie-Hellman
Glossary 307
Preshared Key
309
Numerics
310
311
312
313
314