Re: Re: Flawed Surveys [was: VPN endpoints] (Paul D. Robertson)

[Abe Singer <abe () sdsc edu>]

On Wed, Sep 01, 2004 at 09:22:27PM -0400, Marcus J. Ranum wrote:
> Abe Singer wrote:
> > How about instead of continuing the "my idea is less f*ck3d than
> > *your* idea, there be a more productive discussion of what some good
> > methodologies would be for identifying, collecting, and analysing data
> > to produce metrics.
>
> Well, that's all in a Stats 101 textbook, or any good book on
> testing methodologies and statistics. That's the whole point:
> there is no need to reinvent this particular wheel wrong. It's
> been done; it's taught in most social sciences and math
> curricula at virtually any university.

I know it's all in Stats 101, what I meant was, assuming we've all read
Stats 101, let's talk about these things in the context of computer
security.  How about some proposed sound methodologies for measuring of
security thingamabobs?

What are some hypotheses about computer security?  How should we go about
testing them?  What kind of metrics would we like to see?   Which methods
in Stats 101 should we use to compute them?  What data do we need to
compute them?  Where and how do we get the data?

The answers are *not* in Stats 101, cuz the answers are specific to 
computer security.  Other "sciences" have healthy debates over these
questions wrt their own fields.  Let's do the same!

> > * If you are going to do a survey, how do you target/vet respondents?
> > What questions do you ask.  What controls do you have in place?
>
> Read any Stats 101 or experimental methods textbook. The
> reference I posted earlier on research methods (ISBN: 0767421523)
> has an excellent overview of the process.

Again, I did not mean the general process, I meant in the context of
computer security.  What *specific* questions would you propose asking?
How do you word them to get more accurate answers?  Who specifically would
you target for the survey?  How do you go about getting a representative
sample?  What's the acceptible sample size?  Etc.


> All the things you ask are covered in any introductory texts
> on research and/or statistics. Really. We don't need to go into
> it here! :)

No they're not, see above. :-)

> > It *would* be really useful to have some truly meaningful measurements.
> > It could do a lot to reduce the amount of snake-oil and magic security dust
> > beings sold.
>
> YEAH!
>
> I think the main point everyone seems to want to ignore is the most
> important one I made in my original posting:
> It's NOT MUCH HARDER TO DO IT RIGHT - it just takes a little

Well, in some cases it may be significantly more effort than sending
out a bubble-form and saying "please mail this back to us, we'll be your
best friend if you do, and we even included the stamp!"

But, the other problem is that it's just not sexy or fun.  A lot of this
type of work is drudgery -- looking up data, putting it into tables,
normalizing the data, doing some math, etc.  Not nearly as much fun as
building a skin an MP3 player, or yet another log parser, or setting up
a blog on the web server, or installing linux on a c64...

> bit of learning and some willingness to not charge straight in and
> start calculating the standard deviation of some bullsh&t.  There's
> that old chestnut about how Computer "Scientists" have to re-invent
> the wheel every time because they're a bunch of immature jerks.
> I guess what I am saying is that it *appears* in this case (modulo
> sampling bias!) to be true - rather than learn statistics from a book,
> *EVERY* *SINGLE* security-related survey I have ever seen
> has significant methodological flaws. Are you guys comfortable
> being part of an industry that is somewhere between "witch doctor"
> and "cargo cult" on the spectrum of intellectual integrity?? I'm not!

This is basically where the medical profession was about 100 years ago.
Medicines used to be hawked claiming to cure all sorts of ills with absolutely
no clinical testing that showed any evidence of efficacy.  Doctors could
hang out shingles without any sort of license that showed a minimun of education,
etc.

Clinical testing, drug trials, medical licensing, use of medical histories
and statistics has all developed in the last century.  And it's not perfect,
but it's a lot better than it was.  We at least know that aspirin often helps
relieve headaches.

It's not that much longer that the notion of "scientific method" evolved.
150-200 years ago it was not uncommon to make up data, "correct" data
that didn't fit the theory, or throw out data that didn't fit.



_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards