Re: Security Policy methodologies

[Ted Doty <ted () iss net>]

At 01:08 AM 1/3/98 -0600, Aleph One wrote:
> On Fri, 2 Jan 1998, Ted Doty wrote:
>
> > Until there is a better body of collected (and published) evidence on
> > network attack that can be used to establish norms of security practice,
> > the approach that Hanscom took is likely to be about as good as we can
get. 
> I suggest you read John D. Howard's PhD dissertation, "An Analysis Of
> Security Incidents On The Internet 1989 - 1995". It has some solid
> statistics and trend analysis on the incidents reported to CERT during
> that time period. It also contains an interesting attack taxonomy. But at
> about 300 pages it is not a one night read.

I have indeed read Dr. Howard's dissertation.  While it is a rigorous
analysis of the existing (reported) CERT incidents, it is flawed for
several reasons:

1.  It is only based on incidents reported to CERT.  It seems that there
are many organizations who do not report incidents.  Sometimes this is due
to corporate policy (don't hang out the dirty laundry); sometimes it's due
to a perception in some people that CERT collects information without
helping incident recovery much.  This is unfortunate, and does not really
represent the actual situation with CERT, but does seem to occur - at least
I know of people who "self select" themselves out of this dataset, by not
reporting incidents.

2.  It is only based on incidents that were detected (duh!).  If it is true
that the majority of incidents are not detected, then the results can
easily become skewed by one or two orders of magnitude.  Dan Farmer
suggests this, as does the US General Accounting Office.  Farmer's survey
can be found at http://www.trouble.org/survey/; and the GAO report is at
http://www.gao.gov/AIndexFY96/abstracts/ai96084.htm.

So while Dr. Howard's dissertation contains a wealth of data, it is not at
all clear how applicable this data is to the majority of organizations on
the Internet.  Certainly his conclusion (that the liklihood of a domain
being attacked is between once every 15 years and once in 0.8 years)
contradicts the experience of people I know - one of whom had 25 major
(i.e. root level) incidents in 1997.  OBTW, this person is someone I
consider to be competant and consciencious, and who seemingly has
management support.  None of these incidents were reported to CERT.

Personally, I wish that Dr. Howard had included a section in his otherwise
impressive dissertation on the statistical validity of the data.

Anyone interested in reading the dissertation, you can find it at
http://www.cert.org/research/JHThesis/index.html

- Ted

--------------------------------------------------------------
Ted Doty, Internet Security Systems | Phone: +1 770 395 0150
41 Perimeter Center East            | Fax:   +1 770 395 1972
Atlanta, GA 30346  USA              | Web: http://www.iss.net
--------------------------------------------------------------
PGP key fingerprint: 362A EAC7 9E08 1689  FD0F E625 D525 E1BE