Re: Flawed Surveys [was: VPN endpoints] (Paul D. Robertson)

[Abe Singer <abe () sdsc edu>]

> Date: Wed, 1 Sep 2004 16:16:38 -0400 (EDT)
> From: "Paul D. Robertson" <paul () compuwar net>
>
> I'm not saying "Let's base everything we can on surveys!"  I'm saying that
> survey data can be useful, and you can improve the usefulness of that data
> by throwing out the obviously bad data (ooutliers) and by checking against
> the data you do have.


How about instead of continuing the "my idea is less f*ck3d than
*your* idea, there be a more productive discussion of what some good
methodologies would be for identifying, collecting, and analysing data
to produce metrics.

* If you are going to do a survey, how do you target/vet respondents?
What questions do you ask.  What controls do you have in place?

* If you collect incident data, financial data, etc., what data, and how
do you validate it?

* What do you do with all this data once you collect it? What sort of
analysis?  How do you calculate amount of error?  How do you account
for missing data?  How do you interpret the results of your analysis?

Maybe you'll never get the data you need, or it will cost to much to get it,
but you won't really know that until you can say what it actually is.

It *would* be really useful to have some truly meaningful measurements.
It could do a lot to reduce the amount of snake-oil and magic security dust
beings sold.
_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards