Re: VPN endpoints

["Paul D. Robertson" <paul () compuwar net>]

On Mon, 30 Aug 2004, Marcus J. Ranum wrote:

> or the CIO magazine survey on security) - a lot of these surveys are
> fundamentally flawed. They yield results but it's hard to say what the
> results actually _measured_.

So long as they're flawed approximately the same way from survey to
survey, they're often both "better than nothing[1]" and a good relative
metric.  We often don't need absolute metrics, relative metrics will do
just fine.  I know what my $foo risk was last year, and I know what it was
the year before, and I can compare to the survey and see the relative
differences and the relative change- therefore, I can figure out my
approximate relative change for this year.

> Specifically, many security surveys are based on self-selected
> samples (e.g: "polls"). When you do a poll, what you're doing is
> asking "Please fill this out."   But there are a lot of assumptions
> that get dropped on the floor. :(  What you're really measuring is:
>         - How much the person cared about the topic (motive to respond)
>         - How honest the respondent is (hard to verify)
>         - Other factors (hard to predict)

You can also (a) drop outliers, (b) have cross-conflicting questions, and
(c) answer the questions on behalf of a known quantity and still be able
to validate polls pretty well.  You obviously don't get people who don't
care to respond, but if the number of people who do respond is
significant, that's ok.

> I'm sure nobody on this list has ever filled out one of those surveys
> from a magazine in which they asked you your job position, whether
> you were a decision-maker, company size, etc...  And I'm sure you
> all fill them out EXACTLY right. I used to enjoy periodically asserting
> that I was the CEO of a 1 person company with a $4,000,000 IT
> budget (well, a guy can dream, huh?)     Unfortunately, sometimes

You're out of the range of the mean by orders of magnitude, anyone doing
it even half-way should be throwing that response away (assuming they
*want* correct data,) which in that case is only half-right- better
qualified leads should be worth more, but either fudging is built into the
pricing model, you got sold cheaper, they didn't care, or someone got
ripped off.

Paul
[1] That doesn't mean they aren't often worse than nothing, just that they
can be useful.  Just like assessing risk actually- same rules apply.
-----------------------------------------------------------------------------
Paul D. Robertson      "My statements in this message are personal opinions
paul () compuwar net       which may have no basis whatsoever in fact."
probertson () trusecure com Director of Risk Assessment TruSecure Corporation
_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards