Troubleshooting, 240 | HP Host Intrusion Detection System (HIDS)

Troubleshooting

Troubleshooting

Troubleshooting

This section describes a variety of potential problems and their solutions. To stay current with product updates and patches, be sure to monitor the HP security software news and events web site at www.hp.com/security.

Agent and System Manager cannot communicate with each other

(No errors are being generated by the HP-UX HIDS processes and everything seems to be running ﬁne otherwise.) See also “No Agent Available” on page 246.

❏Make sure the check sums on the following two ﬁles are identical: On the Administration system, run:

/usr/bin/cksum /etc/opt/ids/certs/admin/cacert.pem

On the Agent system, run:

/usr/bin/cksum /etc/opt/ids/certs/agent/cacert.pem

If the results are different, the Administration and Agent certiﬁcates are not signed by the same trusted certiﬁcate authority and the communication handshake fails.

To correct this, regenerate the agent certiﬁcate and install the new certiﬁcate bundle on the agent system. Recheck the check sums to verify that they match.

❏If there is a ﬁrewall positioned between the administration system and an agent system, be sure the HP-UX HIDS ports are enabled. See “Working with Firewalls” on page 30.

❏If you are using NIS on these systems, it is likely the port information is not being retrieved from your /etc/services ﬁle.

•Inform the NIS Master about the HP-UX HIDS ports. See “Working with NIS” on page 30.

•Launch the System Manager and verify that the agent is now available.

❏Increase the response timeout in the Preferences screen. See “General Preferences” on page 116.

❏Increase the value for IDS_SSL_TIMEOUT in the agent conﬁguration ﬁle, ids.cf. See “Remote Communication Conﬁguration” on page 209.

❏If the agent system is multihomed, make sure the agent and administration systems are properly conﬁgured. See “Conﬁguring a Multihomed Agent System” on page 25.

❏If the administration system is multihomed, make sure the agent and administration systems are properly conﬁgured. See “Conﬁguring a Multihomed Administration System” on page 27.

240	Appendix G

Image 252

HP Host Intrusion Detection System (HIDS) Troubleshooting, Agent and System Manager cannot communicate with each other

Contents

HP-UX Host Intrusion Detection System Administrator’s Guide EditionManufacturing Part Number J5083-90013 December Warranty Government License Trademarks Iii Conventions Contents System Manager Screen Schedule Manager Screen Host Manager Screen Network Node ScreenVii Preferences Screen Templates and AlertsViii Idsagent Command Idsadmin CommandAutomated Response Agent Conﬁguration File Troubleshooting MessagesHP Software License Original SSLeay License HP Software License Terms Xii Overview Summary Documentation Why Do You Need Intrusion Detection? Loss of Financial AssetsLoss of Intellectual Property Loss of Computing Resources Who Are the Perpetrators? How Are These Threats Realized?Misplaced Trust Malicious Code Why Existing Tools Are Only Part of the Solution Excessive Privilege for Simple TasksBeing Used as a Springboard to Attack the Next Victim Firewalls Encryption Security Auditing Tools Where Does Intrusion Detection Fit In? What Is Intrusion Detection? What HP-UX Hids Does What HP-UX Hids Does Not Do HP-UX Hids Components Graphic Representation How the Components Interact to Detect Intrusions HP-UX Hids Components HP-UX Hids Secure Communications Detection TemplatesSurveillance Groups Surveillance Schedules Glossary of HP-UX Hids Terms Intrusion Detection Data Intrusion Detection SystemKernel Node System Manager VirusVulnerability Glossary of HP-UX Hids Terms Chapter Conﬁguration Conﬁguration Introduction RequiredOptional Setting Up the HP-UX Hids Secure Communications Overview of Procedures to Set Up Secure CommunicationsScript to Use Where Used End Product Create the X.509 Certiﬁcates $ IDSgenAdminKeys install $ IDSgenAgentCerts TIP Transport the Certiﬁcates Install the Keys on Each Host $ IDSimportAgentKeys /var/opt/ids/tmp/myhost1.tar.Z myadmin Conﬁguring a Multihomed Agent System Step Example $ nslookup large2 Conﬁguring a Multihomed Administration System To conﬁgure a multihomed administration system Edit the agent conﬁguration ﬁle for example Conﬁguring a Loopback System To conﬁgure a loopback system Conﬁguring Ports Working with NISWorking with Firewalls Enabling Large Numbers of Agents Enabling Over 23 Agents Thread LimitsTo change the value of maxthreadproc Select Kernel Conﬁguration Select Conﬁgurable Parameters Enabling Over 20 Inbound Requests To view and change the value of tcpconnrequestmax Accessing Manpages Restricting PermissionsRuntime File Permissions Files Permissions Accessing Manpages Chapter Getting Started Getting Started Agents System Manager Set up hosts and run schedules Starting HP-UX Hids for the First Time See , Host Manager Screen, on Operations Screens Schedule ManagerHost Manager Network Node Basic Screen Actions Selecting Entries in ListsSearching Entries Sorting Entries Basic Screen Actions Chapter System Manager Screen System Manager Screen System Manager Screen Starting the HP-UX Hids System Manager Stopping the HP-UX Hids System ManagerTo start the HP-UX Hids System Manager To stop the HP-UX Hids System Manager On the System Manager Screen Status Field ValuesStatus Value Description Getting the Status of Agent Hosts To get the status of agent hostsOn the System Manager screen Resynchronizing Agent Hosts To resynchronize agent hosts Activating a Schedule on Agent Hosts To activate a surveillance schedule on agent hostsChoose the Actions Activate Schedule menu item Stopping Schedules on Agent Hosts To stop a surveillance schedule on agent hosts Starting HP-UX Hids Agents To start the agent Halting HP-UX Hids Agents To halt agents remotely from the System ManagerTo halt the agent locally on the agent host Accessing Other Screens Go to Schedule Manager ScreenGo to Host Manager Screen To go to the Schedule Manager screen Go to Network Node Screen Go to Preferences ScreenReturn to System Manager Screen Accessing Other Screens Chapter Schedule Manager Screen Schedule Manager Screen Schedule Manager Creating a Surveillance Schedule To create a surveillance schedule Displaying the Schedule Manager Screen Closing the Schedule Manager ScreenTo display the Schedule Manager screen To close the Schedule Manager screen Conﬁguring Surveillance Schedules Creating a New Surveillance ScheduleCopying a Surveillance Schedule To create a new surveillance schedule Modifying a Surveillance Schedule To modify a surveillance schedule Renaming a Surveillance Schedule To rename a surveillance scheduleChoose File Save Selected Schedule As Deleting a Surveillance Schedule Undoing and Redoing ChangesTo delete a surveillance schedule Saving a Surveillance Schedule To save a surveillance scheduleChoose File Save Selected Schedule Conﬁguring Surveillance Groups Creating a New Surveillance GroupCopying a Surveillance Group To create a new surveillance group Modifying a Surveillance Group To modify a surveillance group Renaming a Surveillance Group Rename Surveillance Group DialogTo rename a surveillance group Deleting a Surveillance Group Saving a Surveillance GroupTo delete a surveillance group Conﬁguring Detection Templates Modifying a Property Value In a TemplateTo change the value of a property in a detection template Edit List Dialog To add a new value Suggested Best Practices 11Edit Dialog Edit Some Template Conﬁguration Guidelines Setting Surveillance Schedule Timetables Specifying When a Schedule Will Run To specify when a schedule will run Canceling Changes See Saving a Surveillance Schedule on Viewing Surveillance Schedule Details Viewing the Source of a Surveillance ScheduleRefreshing the Details Display To view the source of a surveillance schedule Clearing the Details Display Saving the Details DisplaySave Dialog To clear the display Predeﬁned Surveillance Schedules and Groups Predeﬁned Surveillance Schedules Predeﬁned Surveillance Schedules Host Manager Screen Host Manager Screen Managing Hosts Displaying the Host Manager Screen Closing the Host Manager Screen Adding New Hosts Adding a New Host ManuallyAdd Host Dialog To add a new host manually Host Name Address ﬁeldIP Address Adding New Hosts from /etc/hosts Name ﬁeldHost Name and IP Address To add new hosts from /etc/hosts Adding New Hosts from a File Rules for Host Lists FilesOpen Dialog To add new hosts from a ﬁle Modifying a Host To modify a host entry Deleting Hosts To delete a host entry To enable or disable an agent host for monitoring Enabling and Disabling Hosts Managing Tags Add, modify or delete tags To add a tag To edit a tag To delete a tag Maintaining Host Files Saving the Host List in the Current FileSaving the Host List in a Different File Using an Alternate Host List File Using Multiple Host Files Maintaining Host Files Chapter Network Node Screen 100 Network Node Screen Opening a Network Node ScreenClosing a Network Node Screen To display the Network Node screen for an agent host Alerts Tab 102 HP-UX Hids Alerts What They Mean, What to Do Errors Tab HP-UX Hids Errors What They Mean, What to Do104 General Operations Selecting EntriesSelecting with the Mouse Simple Version Searching for the Next Unseen Entry Searching for a StringFind Dialog To delete one or more alerts or errors Deleting an EntryMarking Entries as Seen or Unseen To search again Unseen 108 Saving a Log File Set Network Node screen from the System Manager screenSaving the Current Log File Set Saving a New Log File Set Save Dialog Box Press Ctrl-AExample Creating a New File Set Example Saving the File Set over Another File Set Opening a Log File Set Log File RotationOpen Dialog Box 112 Preferences Screen 114 Preferences Screen Option Default Description General PreferencesTo choosing Actions Status Poll from the System Manager 116 Actions Resync from the System Manager screen Column Name Default Description Browser PreferencesAlert Events Preferences 118 Error Events Preferences Column Default Description Name System Manager Subtab 120 Templates and Alerts Alerts LimitationsProperty Types Templates Alert Summary Table A-1 Detection TemplatesAttack Detected Alert Alert Severity Detection Template 124 Appendix a 125 Unix Regular Expressions Examples126 Appendix a 127 Limitations 128 Template Property Types Type I Pathnames to Not Monitor Type II Pathnames/Programs Pairs 130 Type III UIDs Type IV UID Pairs Type V Network Triplets Type VI Time Strings132 Type VII Flags Type Viii Scalars Buffer Overﬂow Template 134 Name Type Default Value Execute on StackTable A-2 Template Properties Table A-3 Execute on Stack Alert Properties 136 Unusual Argument Length Table A-4 Unusual Argument Length Alert Properties Argument with Non-printable Character 138 Appendix a 139 140 Race Condition Template Table A-6 Template Properties File Reference Modiﬁcation Table A-7 File Reference Modiﬁcation Alert Properties142 Appendix a 143 Privileged Setuid Script Executed Table A-8 Setuid Script Executed Alert Properties144 Appendix a 145 Modiﬁcation of Files/Directories Template Table A-9 Template Properties146 Properties 148 File Being Modiﬁed Table A-10 File Being Modiﬁed Alert Properties 150 Appendix a 151 Changes to Log File Template Table A-11 Template Properties152 Append-Only File Being Modiﬁed Table A-12 Append-Only File Being Modiﬁed Alert Properties 154 Creation of Setuid File Template Table A-13 Template PropertiesAlerts generated By this template Setuid File Created Table A-14 Setuid File Created Alert Properties156 Appendix a 157 Creation of World-Writable File Template Table A-15 Template Properties158 World-Writable File Created Table A-16 World-writable File Created Alert Properties 160 Appendix a 161 162 Modiﬁcation of Another User’s File Template Table A-17 Template Properties Non-owned File Being Modiﬁed Table A-18 Non-owned File Being Modiﬁed Alert Properties164 Appendix a 165 Limitations 166 Login/Logout Template Table A-19 Template Properties 168 Login/Logout Table A-20 Login/Logout Alert Properties Successful su Detected Table A-21 Successful su Detected Alert Properties170 Appendix a 171 172 Repeated Failed Logins Template Table A-22 Template PropertiesTemplate How this template Failed Login Attempts Table A-23 Failed Login Attempts Alert Properties174 Appendix a 175 Repeated Failed su Commands Template Repeated Failed su AttemptsTable A-24 Template Properties Table A-25 Repeated Failed Su Attempts Alert Properties Appendix a 177 Template Conﬁguration Syntax 178 Appendix a 179 180 Automated Response 182 General Guidelines Response Methods 184 How Automated Response Works in HP-UX Hids Alert ProcessSecurity checks Programming Notes Table B-1 Additional Arguments Passed to Response Programs 186 Appendix B 187 Table B-3 Environment Variables Set for Response Programs Name Value Description188 Appendix B 189 Programming Guidelines Writing Perl vs. Shell Response ScriptsWriting Privileged Response Programs 190 Solution a Code ExamplesCode for scriptA.sh Solution B Code for privA programCode for PrivB program 192 Solution C Code for privC program Code for scriptC.sh script #!/usr/bin/sh194 Sample Response Programs Sample C Language Program Source CodeSample Shell Script Alert Responses Forwarding Information 196 Appendix B 197 Halting any further attacks 198 Appendix B 199 Preservation of evidence 200 Appendix B 201 Restoration of a known good state 202 OVO Enablement in HP-UX Hids HP OpenView Operations Smart Plug-In 204 Idsagent Command 206 Idsagent Command Synopsis Options Example 208 Idsadmin Command 210 Idsadmin Command Synopsis Startup Options Commands 212 Agent Conﬁguration File 214 Agent Conﬁguration File Forcing Active Agent to Reread Conﬁguration File Name Default Value Global ConﬁgurationTable E-1 Global Conﬁguration Variables 216 Data Source Process Conﬁguration Kernel Audit Data DSPTable E-2 DSP idskernDSP Parameters 218 Remote Communication Conﬁguration Table E-3Correlator Conﬁguration Variables 220 Messages 222 Agent Messages Idsagent failed to reopen stderr in append mode Idsagent internal error in handling signature groupsIdsagent failed to initialize conﬁguration module Idsagent failed to start group Idsagent unable to setup Sigchld signal handler Idsagent unable to setup Sighup signal handlerIdsagent unable to setup signal handler Idsagent unable to setup Sigsegv signal handler Idsagent error trying to shutdown a process Idsagent failed to allocate memoryIdsagent failed to create schedule path ﬁlename Idsagent failed to execute correlator corr Idsagent internal error no correlator in PMStartProcesses Idsagent internal error occurred in PMStopGroupIdsagent failed to initialize schedule Idsagent failed to initialize schedule in crontab Idsagent not enough disk space to create schedule Idsagent not enough disk space to parse scheduleIdsagent not enough disk space to save conﬁg ﬁle Idsagent out of process table space Internal error Internal error unknown stateUnable to open the response script directory dir System Manager Messages Exception while opening ﬁle filename File Save Error Incomplete or Invalid Entry Data Entry ErrorInvalid Host State Unable to disable host Invalid Property Value value Property Value Error No more instances of searchstring found Find Error Only one property may be edited at a time Selection ErrorSearchstring not found Find Error Select Property to be edited Selection Error Select Surveillance Group Name to delete Selection Error Select Surveillance Group to copy Selection ErrorSelect Surveillance Schedule to copy Selection Error Select Surveillance Schedule to delete Selection Error Surveillance Schedule not selected Schedule Selection Error Following hosts are in an invalid state for this commandUnable to Overwrite filename File Save Error 234 Unknown Host unable to resolve IP Address IPaddress Unknown IP Address unable to resolve Host Name 236 Troubleshooting 238 Appendix G 239 Troubleshooting Agent and System Manager cannot communicate with each other240 Agent does not start on system boot $ /usr/sbin/kmtune -q enableidds Agent needs further troubleshooting To clean up the IDS message queuesAgent host appears to hang and/or you see message disk full 242 Agent does not start after installation Agents appear to be stuck in polling statusAlert date/time sort seems inconsistent Alerts are not being displayed in the alert browser Idsadmin needs installed agent certiﬁcates Buffer overﬂow triggers false positivesDuplicate alerts appear in System Manager 244 IDScheckInstall fails with a kmtune error IDSgenAdminKeys or idsgui quits early Large ﬁles in /var/opt/ids Log ﬁles are ﬁlling upNo Agent Available 246 SSH does not perform a clean exit after idsgent is started Schedule Manager timetable screen appears to hang System Manager appears to hang System Manager does not start after idsgui is started248 Using HP-UX Hids with IPFilter and SecureShell Unknown program and arguments in certain alert messagesIPFilter rules for HP-UX Hids How to allow the SecureShell daemon to forward X11 trafﬁc 250 Appendix G 251 252 HP Software License Appendix H 253 OpenSSL License 254 Original SSLeay License Appendix H 255 256 HP Software License Terms 258