Re: StackGuard with ... Re: [Paper] Format bugs.

[Keith Owens <kaos () OCS COM AU>]

On Sat, 22 Jul 2000 11:43:26 -0400,
Alan DeKok <aland () STRIKER OTTAWA ON CA> wrote:
>  The problem (in the end) is NOT printf and friends, it's <stdarg.h>,
> and the va_*() functions.  They can trivially be convinced to walk up
> the stack, returning arguments which were not passed to the function.
> This is a bug in the C compiler.
> [snip]
>  va_arg() should be a built-in function, which ensures that it
> doesn't return more arguments than were passed.  It should be possible
> to add these checks to the gcc, as the compiler knows the size of the
> current stack frame.

The size of the current stack frame is irrelevant, it is the previous
stack frame that matters and gcc does not know its size.  On ix86, the
stack looks like this on entry to a function.

  +-----------------------------+<--.
  | Locals and automatics,      |   |
  | frame 1. (variable size)    |   |
  +-----------------------------+   |
  | Save registers,             |   |
  | frame 1. (variable size)    |   |
  +-----------------------------+   |
  | Arguments to frame 0,       |   |
  | (variable size)             |   |
  +-----------------------------+   |
  | Return address, frame 1     |   |
  +-----------------------------+   |  <=== sp on entry
  | Frame Pointer, frame 1      |>--'
  +-----------------------------+
  | Locals and automatics,      |
  | frame 0. (variable size)    |
  +-----------------------------+
  | Save registers,             |
  | frame 0. (variable size)    |
  +-----------------------------+      <=== current sp

gcc knows the size of the current stack frame which runs from (sp on
entry) to (current sp).  But the parameters to the current function are
in the previous stack frame and gcc has no guaranteed way of getting
the size of that frame, therefore it cannot verify the number of
parameters that were passed nor can it prevent va_arg() from scanning
"too far" up the previous frame.

In an ideal world the "frame pointer" field would link each frame
together so gcc could find the start of the previous frame.  But
programs can be compiled with -fomit-frame-pointer and their frames
will not contain a frame pointer.

If the va_arg() function is compiled with frame pointers it will save
ix86 %ebp on entry as a frame pointer.  However if the calling function
was compiled with -fomit-frame-pointer then the value of %ebp on entry
to the va_arg() function is garbage.  Optional frame pointers are
inherently unreliable.

The bottom line - on architectures that do not mandate a frame pointer
or its equivalent, gcc has no way of getting the size of the calling
frame.  This is one of the reasons that gdb has so many problems doing
a backtrace on functions compiled with -fomit-frame-pointer.