operator new[] overflow checking in G++

[Florian Weimer <fweimer () redhat com>]

Last week, I was finally able to fix the operator new[] overflow in gcc 
trunk:

  <http://gcc.gnu.org/bugzilla/show_bug.cgi?id=19351>

This version rejects any attempt to allocate an array of variable-length 
arrays, thus avoiding a variable-times-variable multiplication, for 
which the overflow check was deemed too costly in previous discussions 
with GCC developers.  Variable-length arrays are a GCC extension carried 
over from C++ mode.

The nature of this fix requires that affected software is recompiled—the 
multiplication which needs checking is inside code generated by the 
compiler.  Looking exclusively at referenced symbols, it is not possible 
to check if a C++ program uses operator new[].  (Most don't, std::vector 
is not affected.)

There is another patch which touches code which is not actually used by 
GCC, but could theoretically be called by code emitted by other compilers:

  <http://gcc.gnu.org/ml/gcc-patches/2012-08/msg01416.html>

We're working on a backport of the patch to GCC 4.7:

  <https://bugzilla.redhat.com/show_bug.cgi?id=850911>

This patch will not reject previously accepted programs.  The current 
version does not check for overflow in the array-of-VLAs case, but we 
might still change this.

Additional testing for both versions of the patch is welcome.

PS: If this receives a CVE, it will need one from 2002:
<http://cert.uni-stuttgart.de/ticker/advisories/calloc.html>
--
Florian Weimer / Red Hat Product Security Team