breakpoint the stack buffer overflow from executing malicious code like SQL Slammer worm

["Peter Huang" <yinrong () rogers com>]

Hi, David and others:

For the past few days, I have done a few experiments and some research on
ways to prevent the buffer overflow exploitation. I believe the following
compiler option (if implemented and used) should make the exploitation of
stack buffer overflow by "jmp esp" method impossible (as far as I know) to
execute a piece of malicious code on Intel-Inside PC. The buffer
vulnerability, if exists, still overflows to be trapped by this mechanism if
being pumped too much.

For a function like:

Void testVoid (void)
{
        do something here and the stack overflows such as sprintf in ssnetlib.dll
in SQL 2000 server (pre-SP3)
}

The compiler inserts the following opcodes before "ret".

Mov     ecx, [esp]
Cmp     word ptr [ecx], 0FFE4h  ; check for "jmp esp"
Jnz     realRet
Mov     byte ptr [esp+4], 0cch  ; stack has been overflowed and function
; will breakpoint (no need to go on anyway)
RealRet:
        ret

For a function like:

Void testParametered(int I, int i2, int i3 ...)
{
        do something here and the stack overflows such as sprintf in ssnetlib.dll
in SQL 2000 server (pre-SP3)
}

The compiler inserts the following opcode before "ret"
        Mov     byte ptr[esp+4], 0cch   ; if return address is not overwritten,
                                    ; then this will be discarded by opcode such
as
                                    ; "add esp, xx" down the execution path.
                                    ; Otherwise, the exploitation code injected
onto the stack,
                                    ; which happens before this move, will
                                    ; cause a breakpoint trap to happen due to the
                                    ; "jmp esp" method exploited by the buffer
overflow.
        Ret

Normally, both above functions continue normally. However, if the stack is
overflowed due to some software bugs under exploitation attack, then the
mechanism listed above will generate breakpoint trap and the execution of
malicious code is stopped.

Unfortunately, it will definitely increase the executable image size and
some runtime hit (should not be much because of the cache lines are still
valid etc.). Some might think that this creates some false illusion of
security. However, it does dam the overflow to what has been overflowed and
prevent the overflow from spreading so that we do not have to hear others
say:" dam it".

Here is what I got:


Best regards

Peter Huang
http://members.rogers.com/yinrong/articles/Prevent.htm