Re: Sardonix Security Auditing Portal

[Paul Robertson <proberts () patriot net>]

On Tue, 5 Feb 2002, Crispin Cowan wrote:

> Since then, LSAP has failed to really live up to its mission.  While the
> LSAP mailing list has become a very nice chat room for discussing
> security issues, not much software is actually audited any more.

Might that be because the LSAP hasn't encouraged new auditors- without a
"how do I audit code?" section, people who aren't accomplished in
whichever language it is won't begin in a place where they could learn and
be useful.

> We propose to address this under used potential by providing a real &
> effective web portal to facilitate & encourage source code auditing.
> This web site will facilitate and encourage source code auditing in the
> following ways:
>
>     * Identify auditors: we don't care if auditors are anonymous, but
>       they should be authentic.  I.e. you can go by a handle if you want
>       to, but you should have a public key, so we know it's you.
>     * Repository for audited code:  importantly, identifying who has
>       audited the code, and what issues were found.

Some sort of independent verification or validation would probably help
significantly.

>     * Todo's:  a list of unaudited code that could use some attention.
>       With source code, so would-be auditors don't have to go find it.
>     * Score keeping:  this part is very important.  There's two kinds of
>       scores to keep:
>           * Auditor's score: each auditor is scored based on the volume
>             of code they have audited, and the number of bugs
>             SUBSEQUENTLY found in code that they declared clean.
>           * Code's score: the trustworthiness of a piece of code is a
>             function of the people who have audited the code.
>
> The score keeping is really the most important part of the web site,
> with two key roles to play:
>
>     * the karma whore effect: we conjecture that a web site that will
>       mechanically compute a number of how l33t you are will attract
>       people to audit code.  Consider how hard people will work just
>       score karma points on Slashdot :-)

How will you ensure that people aren't auditing outdated code for which a
fix already exists?  What about the kharma whoring "I'll create bad code
and you fix it" stuff?

>     * assuring code quality: scoring the code in terms of the number &
>       quality of eyes that have read it will give code consumers a
>       reasonably valid way to determine the level of trust they can put
>       in that code.

It'd be nice if some sort of usage metric were assigned to the code in
question so that people aren't going back and fixing code that hasn't been
run in 5 years by anyone on the planet.

> We also will be encouraging responsible reporting. The audit submission
> form explicitly asks you if you have followed the RFP
> http://www.wiretrip.net/rfp/policy.html and notified the package
> maintainer prior to publishing your findings.

Perhaps getting into escrowing the reports would be useful?

> What we need:  community feedback.
>
>     * Generally:  what do folks think of the model?  What changes, if
>       any, do people think it needs to make it work?
>     * Specifically: the scoring functions are critical, as people will
>       generally act to maximize figures of merit. This can get
>       pathological, e.g. stock brokers churning the market to make
>       commission.  What should the scoring functions look like for
>       auditors? for programs?

I'd focus less on scoring programs until you've got a significant history
of auditing- also how do positive audit results play into this?  Or is
this a fail-only process?

Paul
-----------------------------------------------------------------------------
Paul D. Robertson      "My statements in this message are personal opinions
proberts () patriot net      which may have no basis whatsoever in fact."

_______________________________________________
firewall-wizards mailing list
firewall-wizards () nfr com
http://list.nfr.com/mailman/listinfo/firewall-wizards