HP Host Intrusion Detection System (HIDS) manual Buffer Overflow Template, 134

Templates and Alerts

Buffer Overflow Template

The vulnerability addressed by this template

How this template addresses the vulnerability

NOTE

Buffer Overflow Template

All buffer overflow attacks (e.g., stack smashing, return-into-libc, execute on heap) attempt to overflow a buffer, where the buffer can be a local variable residing on the stack or a dynamically allocated buffer residing on the heap or a global variable residing in the process data segment. See the whitepaper “Stack Buffer Overflow Protection in HP-UX 11i,” available at http://www.docs.hp.com, for a description of buffer overflow attacks on HP-UX. Unusually long program arguments are carefully modified by an attacker to overflow a buffer for which the program does not perform bounds checking. By overflowing the buffer, an attacker can modify the program’s execution flow to execute malicious code and thereby hijack a privileged program. Modifying a program’s execution flow can be accomplished in several ways, including the following:

•Overflowing a buffer on the stack to modify the return address in an activation record

•Overflowing a buffer on the heap to modify a free memory header so that the heap memory allocation code subsequently overwrites a function’s return address.

•Overflowing a buffer in the data segment in order to overwrite an adjacent variable containing a function pointer so that a subsequent dereferencing of the variable results in the execution of malicious code.

The Buffer Overflow (BO) template monitors attack patterns that are indicative of various types of buffer overflow attacks and reports execute-on-stack buffer overflow attacks detected by the HP-UX kernel (starting with HP-UX 11i). The template monitors privileged setuid programs where the effective user id (euid) is not equal to the real user id (ruid) and the euid is one of the user ids specified in the template’s property list of privileged users (e.g., root).

Specifically, the template monitors privileged setuid programs for the following:

•The privileged setuid program was invoked with an unusually long program argument(s).

•The privileged setuid program was invoked with program argument(s) that contain non printable characters (e.g., could be CPU opcodes).

The template also reports when the kernel has detected that a program has attempted to execute on its stack (perhaps as part of a stack buffer overflow attack).

In HP-UX 11i v1 and later, comprehensive stack buffer overflow protection, which uses a combination of highly efficient software and existing memory management hardware, protects against both known and unknown buffer overflow attacks without sacrificing system performance. This protection is managed with the executable_stack tunable kernel parameter. You can allow selected programs to execute from the stack by marking them with the -esoption of the chatr command. See the executable_stack (5) and chatr (1) manpages and the Stack Buffer Overflow Protection in HP-UX 11i white paper, available at http://docs.hp.com

134 Appendix A