code review engagement scoping

[kevin.w.wall at gmail.com (Kevin W. Wall)]

kartik trivedi wrote:
> How do people in this group scope code review engagements? What are some
> of the tools one uses to count the number of lines of code, supporting
> libraries, comments, etc. Is there an umbrella list of issues one
> generally looks for in code reviews? We are talking about open source
> products written in C/CPP
>
> Any help is appreciated

The way my group--an application security team--has scoped it at Qwest is
to count the non-commentary source lines (NCSL) of code to be reviewed
and then divide that by our typical rate R (for us, about 180 NCSL/hr)
and add in about the same amount for preparation time and finally multiply
by the # of people involved. That does not take into account the time to
make any resulting changes and to retest though. That mostly is dependent
on how many issues you find.  If you start keeping stats you can come up
with what works for your team, but you have to have people honestly record
their prep time. (To start with, you may want to collect this anonymously
to encourage honesty.)

Lastly, I'd encourage you to keep to a rate somewhere between 120-250 NCSL/hr
depending on the complexity of the code and the familiarity of the subject
matter by the reviewers.

There were some good statistics kept by the 5ESS team at (then AT&T) Bell Labs
back in the 1980s that found that was the optimal sweet spot for bug discovery
rate. If you are first using a static code analyzer and _only_ looking for
_security_ flaws, you might be able to crank that rate up a bit, but I'd
advise against it to start out. Most people starting out think that they
can inspect code at a rate of 2000-3000 NCSL/hr, but that's just nuts IMO.

Anyhow, take that FWIW. Like almost everything else, YMMV, so try different
things and figure out what works for you.

-kevin
-- 
Kevin W. Wall
"The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents."        -- Nathaniel Borenstein, co-creator of MIME