Re: Re: Flawed Surveys [was: VPN endpoints] (Paul D. Robertson)

["Marcus J. Ranum" <mjr () ranum com>]

Abe Singer wrote:
> How about instead of continuing the "my idea is less f*ck3d than
> *your* idea, there be a more productive discussion of what some good
> methodologies would be for identifying, collecting, and analysing data
> to produce metrics.

Well, that's all in a Stats 101 textbook, or any good book on
testing methodologies and statistics. That's the whole point:
there is no need to reinvent this particular wheel wrong. It's
been done; it's taught in most social sciences and math
curricula at virtually any university.

Normally, I am not one to "appeal to authority" on an argument.
I believe that 100+ years of experience with testing, statistics,
and polling, however, is not something to take lightly. ;)  So
I recommend the Stats 101 texts as a good starting point
which will probably remove the need for further discussion.

> * If you are going to do a survey, how do you target/vet respondents?
> What questions do you ask.  What controls do you have in place?

Read any Stats 101 or experimental methods textbook. The
reference I posted earlier on research methods (ISBN: 0767421523)
has an excellent overview of the process.

[...etc...]
All the things you ask are covered in any introductory texts
on research and/or statistics. Really. We don't need to go into
it here! :)

> It *would* be really useful to have some truly meaningful measurements.
> It could do a lot to reduce the amount of snake-oil and magic security dust
> beings sold.

YEAH!

I think the main point everyone seems to want to ignore is the most
important one I made in my original posting:
It's NOT MUCH HARDER TO DO IT RIGHT - it just takes a little
bit of learning and some willingness to not charge straight in and
start calculating the standard deviation of some bullsh&t.  There's
that old chestnut about how Computer "Scientists" have to re-invent
the wheel every time because they're a bunch of immature jerks.
I guess what I am saying is that it *appears* in this case (modulo
sampling bias!) to be true - rather than learn statistics from a book,
*EVERY* *SINGLE* security-related survey I have ever seen
has significant methodological flaws. Are you guys comfortable
being part of an industry that is somewhere between "witch doctor"
and "cargo cult" on the spectrum of intellectual integrity?? I'm not!

mjr.

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards