Stack overflow blocking in commercial packages

[<Glenn_Everhart () bankone com>]

Gents -

A technique for detecting stack overflow and preventing mischief therefrom
could be devised for a process that maintains stack frames intact. On
x86 it is common this to be done, and procedure enter and leave code
manages the arguments.

If a stack overflow occurs, and some system call occurs before the procedure
returns, a wedge in the system call handling could check the stack to see
if the return chain had overflowed. This would be done by examining the
instruction before the return address on the stack and making sure a
call instruction opcode existed there. Such an opcode would indicate the
return address on the stack was explained. Its absence would (in a well
behaved program) indicate something fishy.

By inspecting the stack at least several calls up, the test could be made
hard to miss. 

I see two issues:
1. If the procedure returns before a syscall is done, the stack frames will
look normal. You'd have to do the check on procedure return or some such
thing, and incur large overhead, to catch that.

2. A procedure that did not maintain call frame traceability would make
the check fail. Any real tool that looked at a process (and had better in any case be able to look at only one process 
or those few that you have
reason to believe will not misbehave) needs a notify-only mode to allow
testing for this. If you have only a bought, binary package you will not
want to just crash it if a stack return address gets mis-identified.


There may be a few other cases that can cause trouble.

Still, the wedge at syscall entry would be compact and the ability to block
some overflows may make it worth having. Anyone know of such a thing?

Glenn Everhart


**********************************************************************
This transmission may contain information that is privileged, confidential and/or exempt from disclosure under 
applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, 
distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If 
you received this transmission in error, please immediately contact the sender and destroy the material in its 
entirety, whether in electronic or hard copy format. Thank you
**********************************************************************