RE: Stack overflow blocking in commercial packages

[<Glenn_Everhart () bankone com>]

A non executable stack is a better way to block overflows, and for that
matter a "canary" on the stack would serve the same purpose. I'd like
something that could be used meanwhile on Solaris or Windows or AIX
and wonder if there are any debug abuses that might let one trap the
return. Checking for procedure call instructions at syscalls, and maybe
inside a clock interrupt triggered bit of code, to catch a few cases where
there are no syscalls between the overflow and the procedure return, could be
done on those, just doesn't give full coverage. Something that would run
on return would do better, but unless there's an efficient way to trap
return instructions I can see no way to get to the info in general. Once
upon a time there were systems where microcode breakpoints could be used
for that sort of thing, but at ruinous overhead.

Sounds like the kind of wedge I was hoping for doesn't exist at the moment
though.

Glenn Everhart


-----Original Message-----
From: exon [mailto:exon () home se]
Sent: Sunday, March 28, 2004 6:01 AM
To: webappsec () securityfocus com
Subject: Re: Stack overflow blocking in commercial packages


Glenn_Everhart () bankone com wrote:
> Gents -
>
> A technique for detecting stack overflow and preventing mischief therefrom
> could be devised for a process that maintains stack frames intact. On
> x86 it is common this to be done, and procedure enter and leave code
> manages the arguments.
>
> If a stack overflow occurs, and some system call occurs before the procedure
> returns, a wedge in the system call handling could check the stack to see
> if the return chain had overflowed. This would be done by examining the
> instruction before the return address on the stack and making sure a
> call instruction opcode existed there. Such an opcode would indicate the
> return address on the stack was explained. Its absence would (in a well
> behaved program) indicate something fishy.
>
> By inspecting the stack at least several calls up, the test could be made
> hard to miss. 
>
> I see two issues:
> 1. If the procedure returns before a syscall is done, the stack frames will
> look normal. You'd have to do the check on procedure return or some such
> thing, and incur large overhead, to catch that.
>
> 2. A procedure that did not maintain call frame traceability would make
> the check fail. Any real tool that looked at a process (and had better 
> in any case be able to look at only one process or those few that you have
> reason to believe will not misbehave) needs a notify-only mode to allow
> testing for this. If you have only a bought, binary package you will not
> want to just crash it if a stack return address gets mis-identified.
>
>
> There may be a few other cases that can cause trouble.
>
> Still, the wedge at syscall entry would be compact and the ability to block
> some overflows may make it worth having. Anyone know of such a thing?

The functionality you're after seem to be a lot like that supplied by 
the kernelpatch provided by Owl (http://www.openwall.com/linux), even 
though it does it a bit differently.

In general, most overflows overwrite the return address to make it point 
back to the stack.
The kernel patch allows an option to make RET instructions pointing to 
the stack illegal, whereas CALL instructions still can point wherever 
they want. It IS possible to circumvent this patch, but not for all 
exploits, and it's not very easy even for those where it is possible to 
do so.
The patch CAN possibly break things, but I haven't encountered any 
problems using it the past 6 months.

> Glenn Everhart

Cheers,
Sourcerer / Andreas Ericsson


**********************************************************************
This transmission may contain information that is privileged, confidential and/or exempt from disclosure under 
applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, 
distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If 
you received this transmission in error, please immediately contact the sender and destroy the material in its 
entirety, whether in electronic or hard copy format. Thank you
**********************************************************************