Manuals / Brands / Computer Equipment / Software / HP / Computer Equipment / Software

HP UX IPSec Software Negotiation Fails (Quick Mode processing failed, QM negotiation timeout), Details, Problem, ISAKMP/MM SA Negotiation Succeeded, IPSec/QM SA

1 318

Download 318 pages, 1.91 Mb

Troubleshooting HP-UX IPSec

Troubleshooting Scenarios

Check for the /var/adm/ipsec/javabeans.txt (VeriSign) or /var/adm/ipsec/.Bsec file (Baltimore).

Details

Check the audit log for messages indicating that the certificate for the local or remote system has expired, has been revoked, or has X.509 encoding errors.

You can also try using preshared keys for primary authentication. You will need to configure the same preshared key on both systems.

Check that you have a certificate for the remote system. As part of the IKE dialog, the remote system should send its certificate to the local system. The IKE daemon stores a copy of the certificate in /var/adm/ipsec/certs.txt (VeriSign) or /var/adm/ipsec/.Bcerts (Baltimore). However, these files are encrypted and can only be viewed with ipsec_mgr. Check the expiration date for the local and remote system certificates.

Check that the /var/adm/ipsec/javabeans.txt file (VeriSign) or the /var/adm/ipsec/.Bsec file (Baltimore) has not been deleted. If the applicable file has been deleted, either restore it from a backup or recreate it by re-importing the certificate.

For VeriSign, check that the entry in the certs.txt file for the local system is complete by using ipsec_mgr to examine the certificates in detail. If you have requested a VeriSign certificate but have not completed the process of importing the certificate into IPSec, you will find an entry in the /var/adm/ipsec/certs.txt or /var/adm/ipsec/.Bcerts file for the local system, but there will be no certificate.

ISAKMP/MM SA Negotiation Succeeded, IPSec/QM SA

Negotiation Fails (Quick Mode processing failed, QM negotiation timeout)

Problem

ISAKMP/MM SA negotiation succeeded, the ISAKMP/MM SA was established, but the IPSec/QM SA negotiation failed.

Chapter 5

177

Contents

HP-UXIPSec version A.02.00 Administrator’s Guide Warranty U.S. Government License Copyright Notice Trademark Notice Contents Preface: About This Document 1. HP-UXIPSec Overview 2. Installing HP-UXIPSec 3. Configuring HP-UXIPSec Page 4. Using Certificates with HP-UXIPSec 5. Troubleshooting HP-UXIPSec Page 6. HP-UXIPSec and IPFilter 7. HP-UXIPSec and HP-UXMobile IPv6 8. HP-UXIPSec and MC/ServiceGuard Page 9. HP-UXIPSec and Linux A. Product Specifications B. Migrating from Previous Versions of HP-UXIPSec C. HP-UXIPSec Configuration Examples Page Page Tables Page Figures Page Intended Audience New and Changed Documentation in This Edition -entity -ike -gateway -tunnel -spd_soft Publishing History Table Publishing History Details Document Operating Systems Typographical Conventions Related Documents HP Encourages Your Comments OpenSSL Copyright Notice NOTE Page Page HP-UXIPSec Overview Page Page Internet Key Exchange (IKE) Manual Keys Authentication Header (AH) Authentication Header (AH) Symmetric key hash shared key hash secret key hash Figure Symmetric Key Authentication HMAC-SHA1 HMAC-MD5 Transport and Tunnel Modes IPv6 AH in Transport Mode Tunnel Mode AH in Tunnel Mode Encapsulating Security Payload (ESP) ESP Encryption Symmetric Key Cryptosystem symmetric key cryptography shared key cryptography Page ESP Encryption in Transport Mode ESP in Tunnel Mode ESP with Authentication and Encryption Authenticated ESP Nested ESP in AH Internet Key Exchange (IKE) Internet Key Exchange (IKE) primary authentication Security Associations (SAs) and IKE Phases Security Association Diffie-Hellman SA Establishment Generating Shared Keys: Diffie-Hellman Diffie-HellmanKey Generation IKE Primary Authentication IKE Preshared Key Authentication preshared key Digital Signatures security certificates Public Key Infrastructure (PKI) Manual Keys Manual Keys HP-UXIPSec Topologies HP-UXIPSec Topologies Host-to-HostTopology Figure 1-11IPSec Host-to-HostTopology Host-to-GatewayTopology Figure 1-12 Host-to-Gateway(VPN) Topology Host-to-HostTunnel Topology Figure 1-13 Host-to-HostTunnel Topology Gateway-to-GatewayTopology Figure 1-14IPSec Gateway-to-GatewayTopology HP-UXIPSec Configuration and Management Features HP-UXIPSec Configuration and Management Features Page Page Installing HP-UXIPSec Page HP-UXIPSec Product Requirements HP-UXIPSec Product Requirements Disk Requirements Security Certificate Configuration Utility Requirements Step 1: Verifying HP-UXIPSec Installation and Configuration Prerequisites Step 1: Verifying HP-UXIPSec Installation and Configuration Prerequisites uname swlist Step 2: Loading the HP-UXIPSec Software Step 2: Loading the HP-UXIPSec Software root swinstall Page Step 3: Setting the HP-UXIPSec Password Step 3: Setting the HP-UXIPSec Password ipsec_admin -newpasswd Re-establishingthe HP-UXIPSec Password Step 4: Completing Post-InstallationMigration Requirements Step 4: Completing Post-Installation Migration Requirements Configuring HP-UXIPSec Page Maximizing Security Maximizing Security Bypass List Strong End System Model ndd -set /dev/ipip_strong_es_model ipsec_config add ipsec_config batch ipsec_config delete ipsec_config show Using a Profile File with a Batch File Profile File Structure Creating a Customized Profile File IPv6 Networks nocommit Argument Configuration Overview •Host IPSec Policies •Tunnel IPSec Policies •IKE Policies •IKE Authentication Records •Start-upoptions gateway IPSec policies Step Page Step 1: Configuring Host IPSec Policies Step 1: Configuring Host IPSec Policies Policy Order and Selection default Host IPSec Policy ipsec_config add host Syntax Acceptable Values: -source and -destination and ip_addr[/prefix[/port_number|service_name]] Default: Range: Table ipsec_config Service Names Service Names Service ipsec_config Service Names (Continued) FTP-CONTROL HTTP-TCP HTTP-UDP NTP ICMP ICMPV6 IGMP Default: ALL CAUTION PASS DISCARD transform_name[/lifetime_seconds[/lifetime_kbytes]] TIP ipsec_config Transforms Transforms Transform Name Description ipsec_config Transforms (Continued) ESP_DES_HMAC_SHA1 ESP_3DES ESP_3DES_HMAC_MD5 ESP_3DES_HMAC_SHA1 -flags flags ipsec_config add host Flags Flags Flag EXCLUSIVE Host IPSec Policy Configuration Examples Page ipsec_config add tunnel Syntax -tsource and -tdestination tunnel_address Default: None Page Page Page -action transform_list transform_name Tunnel IPSec Policy Configuration Example Page Step 3: Configuring IKE Policies Step 3: Configuring IKE Policies ipsec_config add ike Syntax -remote ip_addr[/prefix] -authentication authentication_type -authentication authentication_type PSK RSASIG -hashMD5|SHA1 MD5 SHA1 -encryption encryption_algorithm ipsec_config add IKE Command Examples Step 4: Configuring Preshared Keys Using Authentication Records Step 4: Configuring Preshared Keys Using Authentication Records ipsec_config add auth Remote Multi-homedSystems Configuring IKE ID Information with Preshared Keys not ipsec_config add auth Syntax ip_addr[/prefix] Specifying a subnet address filter and a preshared key allows WARNING you to configure a single preshared key for an entire subnet However, HP strongly recommends that you configure an individual authentication record for each remote system with a Page Step 5: Configuring Certificates Step 5: Configuring Certificates Step 6: Configuring the Bypass List (Local IPv4 Addresses) Step 6: Configuring the Bypass List (Local IPv4 Addresses) Logical Interfaces Example ipsec_config add bypass Syntax add bypass ip_address Bypass Configuration Example Step 7: Verify Batch File Syntax Step 7: Verify Batch File Syntax ipsec_config batch batch_file_name -nocommit Step 8: Committing the Batch File Configuration and Verifying Operation ipsec_config batch batch_file_name ipsec_config show all ipsec_admin -status ipsec_report -cache ipsec_report -all -bypass ipsec_config add startup Syntax -newpasswd Step 10: Creating Backup Copies of the Batch File and Configuration Database Baltimore Configuration Files VeriSign Configuration Files Page Using Certificates with HP-UX IPSec Page Overview Overview Security Certificates and Public Key Cryptography public key cryptography asymmetric key cryptography private key Certificate Revocation List (CRL) Digital Signatures IKE Public Key Distribution Requirements Using VeriSign Certificates Using VeriSign Certificates VeriSign PKI Data Flow VeriSign Certificate Tasks Step 1: Verifying Prerequisites Step 2: Configuring Web Proxy Server Parameters Using a Remote Display Device export DISPLAY=display_device:0.0 Step 3: Registering the Administrator Step 4: Requesting and Receiving Certificates Page Page Using Baltimore Certificates Using Baltimore Certificates Baltimore Certificate Tasks must Step 2: Requesting the Baltimore Certificate Step 3: Configuring the Baltimore Certificate Page Page Page Page Configuring Authentication Records with IKE IDs Configuring Authentication Records with IKE IDs Configuring Authentication Records with Certificate-BasedAuthentication Determining the IPv4 Address in the SubjectAlternativeName VeriSign VeriSign SubjectAlternativeName Baltimore Syntax add auth auth_name -remote ip_addr[/prefix] add auth -ltype local_id_type -ltype -lvalue local_id -rtype remote_id_type -rtype FQDN USER-FQDN X500-DN CN=commonName,O=organization,C=country[,OU=organizationUnit] CN ,O ,C=c [,OU add auth Zebra1 -remote10.20.20.20 -rtypeIPV4 \ -rid10.20.20.20 add auth Zebra2 -remote192.6.2.21 -rtypeIPV4 \ -rid10.20.20.20 add auth Black -remote10.10.10.10 -ltypeIPV4 \ -lid10.20.20.20 Retrieving the Certificate Revocation List (CRL) Retrieving the Certificate Revocation List (CRL) VeriSign Baltimore Manually Retrieving a CRL for VeriSign or Baltimore Page Troubleshooting HP-UXIPSec Page IPSec Operation IPSec Operation Establishing Security Associations (SAs) Security Associations 1.Authenticate Identities 2.Establish ISAKMP/MM SA 3.Establish IPSec/QM SAs Internal Processing Outbound Processing Outbound Data 1.Query the Kernel Policy Engine 2.Query the Policy Manager Daemon 3.Establish an ISAKMP/MM SA 4.Establish IPSec/QM SAs 5.Add IPSec/QM SAs to the Kernel SA Database Inbound Data •AH or ESP Packet Page •Clear Text Packet Establishing Tunnel Security Associations Processing Inbound Tunnel Packets Page Troubleshooting Utilities Overview Troubleshooting Utilities Overview Getting General Information Getting General Information Task Command Getting SA Information Getting SA Information Getting Policy Information (Continued) ipsec_report -hostconfigured ipsec_config show gateway ipsec_report -gateway ipsec_report -gateway[active] Getting Interface Information Getting Interface Information ipsec_report -ip ipsec_report -bypass Viewing and Configuring Audit Information Viewing and Configuring Audit Information ipsec_admin -m[axsize] max_audit_file_size ipsec_config add startup argument_list Enabling and Disabling Tracing Troubleshooting Procedures Troubleshooting Procedures Checking Status ipsec_report -all [-file filename] Page Isolating HP-UXIPSec Problems from Upper-layer Problems ipsec_admin -traceon[ tcp | udp | igmp | all ] Checking Policy Configuration Using ipsec_policy Using ipsec_policy -sa15.1.1.1 -sp65535 -da15.2.2.2 -dp23 -ptcp -dirout telnet Configuring HP-UXIPSec Auditing Audit Level Audit Files and Directory Audit File Size ipsec_admin -m[axsize] max_audit_file_size Dynamically Setting Audit Parameters ipsec_admin [-al audit_level] [-au audit_directory] Configuring Startup Audit Parameters ipsec_config add startup [-autobootON|OFF] [-auditlvl audit_level] [-auditdir audit_directory] [-auditlvl [-auditdir ipsec_report -auditaudit_file Filtering Audit File Output by Entity ipsec_report -auditaudit_file -entity entity_name Reporting Problems Reporting Problems netstat -rn •Output from ipsec_admin -status •Output from ipsec_report -all Page Troubleshooting Scenarios Troubleshooting Scenarios HP-UXIPSec Incorrectly Passes Packets Problem Symptoms Solution HP-UXIPSec Incorrectly Attempts to Encrypt/Authenticate Packets ping linkloop HP-UXIPSec Attempts to Encrypt/Authenticate and Fails ipsec_report -audit file Additional Information ipsec_report -audit /var/adm/ipsec/auditdateinfo.log Main Mode processing failed, MM negotiation timeout) Page ISAKMP Primary Authentication with Preshared Key Fails ipsec_config show auth ISAKMP Primary Authentication Fails with Certificates ipsec Negotiation Fails (Quick Mode processing failed, QM negotiation timeout) Manual Keys Fail local inbound remote outbound local outbound remote inbound SADB_ADD for SPI Page ipsec_admin -start Page Corrupt or Missing Configuration Database Using ipsec_migrate ipsec_admin -stop ipsec_migrate -s old_config_file -d new_config_file ipsec_migrate ipsec_config batch batch_file Autoboot is Not Working Properly Administrator Cannot Get a Local VeriSign Certificate Page Kernel Policy Cache Threshold reached or Kernel Policy Cache Threshold exceeded) ipsec_config add startup -spd_soft spd_soft_limit ipsec_config add startup -spd_hard spd_hard_limit HP-UXIPSec and IPFilter Page IPFilter and IPSec Basics IPFilter and IPSec Basics IPFilter and IPSec IPFilter Scenario One Page IPSec UDP Negotiation IPSec UDP Negotiation IPFilter Scenario Two Page When Traffic Appears to be Blocked When Traffic Appears to be Blocked Scenario Three Allowing Protocol 50 and Protocol 51 Traffic Allowing Protocol 50 and Protocol 51 Traffic Packet with Encrypted TCP Data Packet with IPSec-EncryptedTCP Data Scenario Four IPSec Gateways IPSec Gateways HP-UXIPSec and HP-UXMobile IPv6 Page Mobile Node and Home Address Mobile Node MN) home address home network Home Agents and Basic Operation Mobile IPv6 Basic Operation: Correspondent Node to Mobile Node Mobile IPv6 Basic Operation: Mobile Node to Correspondent Route Optimization Mobile IPv6 Route Optimization Securing Mobile IPv6 with HP-UXIPSec Binding Messages Binding Update Binding Acknowledgement messages Return Routability Messages Between the Home Agent and Mobile Node Prefix Discovery Packets Between the Home Agent and Mobile Node Mobile Prefix Advertisement Payload Packets Routed Through the Home Agent payload Page Understanding Gateway IPSec Policies Gateway IPSec Policies Using Manual Keys Selecting Encryption Keys Using the HP-UXStrong Random Number Generator od -Ax -Nnn /dev/random Troubleshooting Manual Key Problems Page Page -source home_agent_addr -destination mn_home_addr -protocolMH -action transform_name -in and -out manual_key_sa_specification auth_key enc_key Step 2: (Recommended) Securing Return Routability Messages Routed Through the Home Agent Mobile IPv6 Home Test Init and Home Test Packets Step 2A: Step 2B: Step 2C: Step 2A: Return Routability Messages: Configuring the Gateway IPSec Policy for Home Agent Page -tunnel rr_tunnel_name -actionFORWARD -flagsMIPV6 -tunnel rr_tunnel_name ipsec_config add tunnel rr_tunnel_name Page Step 3: (Recommended) Securing Prefix Discovery Messages Between the Home Agent and Mobile Node Page Step 4: (Optional) Securing Payload Packets Routed Through the Home Agent Step 4A: Step 4B: Step 4C: -protocolALL [-priority priority_number] -actionFORWARD -flagsMIPV6 interface_name -tunnel payload_tunnel_name -homeclear Step 4C: Payload Packets: Configuring the Home Agent - Mobile Node Tunnel ipsec_config add tunnel payload_tunnel_name Mobile IPv6 Configuration Example Mobile IPv6 Configuration Example 3ffe::83ff:fef7:1111 3ffe::83ff:fef7:2222 Binding Messages add host mn_2222_binding Gateway IPSec Policy for Home Agent - Correspondent Node Segments add gateway mn2222_rr_to_cn 3ffe::83ff:fef7:2222 0::0 add host mn2222_prefix -protoICMPV6 -pri210 -actionESP_AES128_HMAC_SHA1 \ -flagsMIPV6 (Optional) Payload Messages Routed Through the Home Agent Payload Gateway IPSec Policies Gateway IPSec Policy for Home Agent - Correspondent Node Segments Payload Tunnel IPSec Policy add tunnel mn2222_payload_tunnel 3ffe::83ff:fef7:1111 -protocolALL Batch File Template Batch File Template Page Page Page HP-UXIPSec and MC/ServiceGuard Page cluster failover packages fail over adoptive node MC/ServiceGuard Cluster package addresses clients Using HP-UXIPSec with MC/ServiceGuard Package Clients Not Using HP-UXIPSec A.01.07 or Later Page MC/ServiceGuard Heartbeat Requirement and Recommendation Configuration Steps Page Step 1: Configuring a Common HP-UXIPSec Password Step 1: Configuring a Common HP-UXIPSec Password Step 2: Configuring HP-UXHost IPSec Policies for MC/ServiceGuard Determining MC/ServiceGuard Cluster Information Configuring Host IPSec Policies for Package Addresses Configuring PASS Host IPSec Policies for Heartbeat IP Addresses Source IP Destination Source Address IP Address Configuring Host IPSec Policies for MC/ServiceGuard Quorum Server Cluster Node IPSec Policies for Quorum Server IP Address Quorum Server IPSec Policies Configuring Host IPSec Policies for Remote Command Execution Cluster Node IPSec Policies for Remote Command Execution Page Remote Command Client Host IPSec Policies Configuring Host IPSec Policies for ServiceGuard Manager Cluster Node Host IPSec Policies for ServiceGuard Manager ServiceGuard Manager Host IPSec Policies Protoco Configuring Host IPSec Policies for Cluster Object Manager (COM) Cluster Node Host IPSec Policies for COM COM System Host IPSec Policies Summary: MC/ServiceGuard Port Numbers and Protocols MC/ServiceGuard Port Numbers and Protocols Protocols MC/ServiceGuard Port Numbers and Protocols (Continued) Page Step 3: Configuring HP-UXIPSec IKE policies Step 3: Configuring HP-UXIPSec IKE policies Cluster IKE policies Cluster Client IKE policies Step 4: Configuring Authentication Records for Preshared Keys Step 4: Configuring Authentication Records for Preshared Keys Preshared Key Configuration on Cluster Nodes Preshared Key Configuration on Client Nodes Preshared Keys Configuration on Cluster Nodes Remote IP Address Key Preshared Keys Configuration on Client1 Preshared Keys Configuration on Client2 Page Step 5: Configuring Authentication Records for Certificates Step 5: Configuring Authentication Records for Certificates Authentication Records and IKE ID Information Cluster Node Cluster Clients Page IKE ID Configuration on Cluster Nodes IKE ID Configuration on Client1 and Client2 Step 6: Verifying and Testing the HP-UXIPSec Configuration Step 6: Verifying and Testing the HP-UXIPSec Configuration ipsec_policy -sa15.1.1.1 -da15.2.2.2 ipsec_policy -sa15.1.1.1 -sp65535 -da15.2.2.2 -dp5300 -ptcp Step 7: Configuring HP-UXIPSec Start-upOptions Step 7: Configuring HP-UXIPSec Start-up Options Step 8: Distributing HP-UXIPSec Configuration Files Step 8: Distributing HP-UXIPSec Configuration Files Page Step 9: Configuring MC/ServiceGuard Step 9: Configuring MC/ServiceGuard Cluster Configuration Package Configuration Package Control Script Monitor Script Polling Interval Step 10: Starting HP-UXIPSec and MC/ServiceGuard Step 10: Starting HP-UXIPSec and MC/ServiceGuard cmruncl Adding a Node to a Running Cluster Page HP-UXIPSec and Linux Page Limitations of HP-UXIPSec Interoperating with Linux FreeSwan Limitations of HP-UXIPSec Interoperating with Linux FreeSwan Configuration Example Configuration Example Product Specifications Page IPSec RFCs IPSec RFCs Table A-1 Supported IPSec RFCs RFC Number RFC Title Page Product Restrictions Product Restrictions ISAKMP Limitations IPv4 ICMP Messages IPv6 ICMP Messages HP-UXIPSec Transforms HP-UXIPSec Transforms Comparative Key Lengths Table A-2 AH and ESP Algorithms and Key Lengths Algorithm Encryption Algorithms ESP-DES Linux FreeSwan ESP-DES-HMAC-MD5 ESP-DES-HMAC-SHA1 ESP-NULL-HMAC-MD5 ESP-NULL-HMAC-SHA1 Transform Lifetime Negotiation Migrating from Previous Versions of HP-UXIPSec Page Pre-InstallationMigration Instructions Pre-InstallationMigration Instructions MD5 Version Compatibility ipsec_report -audit audit_file_name [-file output_file_name] [-file Migrating from Versions Prior to A.01.03 -stop Not Re-usingConfiguration Files Post-InstallationMigration Instructions Post-InstallationMigration Instructions Configuration File /usr/sbin/ipsec_migrate -s config_file -d new_config_file ipsec_config add startup -autobooton ipsec_admin -start Page C HP-UXIPSec Configuration Examples Page Example 1: telnet Between Two Systems Example 1: telnet Between Two Systems Apple Configuration Figure C-1 Example 1: telnet AB Figure C-2 Example 1: telnet BA -destination15.2.2.2/32/TELNET -priority20 -actionESP_AES128_HMAC_SHA1 add host telnetBA -source15.1.1.1/32/TELNET \ -destination15.2.2.2 -priority30 -actionESP_AES128_HMAC_SHA1 Page Example 2: Authenticated ESP with Exceptions Example 2: Authenticated ESP with Exceptions Figure C-3 Example 2: Network IPSec Policy with Exceptions Carrot Configuration add host potato -destination193.3.3.3 -priority20 \ -actionESP_AES128_HMAC_SHA1 #to modify the default host policy, you must delete #the existing default policy, then re-addit Policy Priority ipsec_config Batch File Entries add ike potato -remote193.3.3.3 -authenticationpsk add ike 192.1.1_net -remote192.1.1.0/24 \ -authenticationrsasig Authentication Record add auth potato -remote193.3.3.3 Example 3: Host to Gateway Example 3: Host to Gateway Figure C-4 Host to Gateway Configuration Example Blue Configuration Host IPSec Policy Tunnel IPSec Policy -actionESP_DES_HMAC_MD5 add auth torouter -rem16.6.6.6 -pskHello Example 4: Manual Keys Example 4: Manual Keys Dog Configuration Cat Configuration add host rlog_dog_to_cat -destination10.2.2.2 \ -source10.4.4.4/32/RLOGIN Glossary Diffie-Hellman Encryption ESP Filter HMAC Page Page Numerics