Practical Firewall Metrics...Was: INtrusion Detection

[Christopher Nicholls <chrisn () softway com au>]

At 11:34 17/02/98 -0600, Aleph One wrote:
> I would disagree. It is not that we are becoming more closed minded, the
> problem is that there is no way to measure the effectiviness of a security
> solution. There is no measuring stick. NCSA certification is a joke. If we
> where to belive every firewall or IDS vendor their software is as good or
> better than the nexts guy and can protect both the little guy and the
> large banks equaly.

Indeed.

Hmmm... I guess my concern is that there is a great deal of confusion
around as to which FW is best and which is certified and what that all
means... 

Like many on this list I am constantly aware of organisations who just get
FWs ("...yeahh, we got a 'certified' FW..." )on the basis of their own
personal risk/exposure (as little as possible) - without ever coming to
grips with what they are protecting, how they are protecting it and whether
what they chose is suitable for protecting what they have... and as for
ID... welll....

The question has to be asked: Are there any practical metrics for assessing
the quality of a firewall? 

By this I am *not* meaning: Which is better - proxy, screening or stateful?
This automatically decends into highly subjective argument, which - while
entertaining for a while - is hardly edifying.

There is a particularly interesting paper by Marcus Ranum at:

http://www.clark.net/pub/mjr/pubs/fwtest/

which goes a long way in exploring and mapping this difficult terrain.
Marcus concludes that a peer review is possibly the only real way of
properly "certifying" or testing something like a firewall... I won't
paraphrase it any more - go and read it for yourselves...

What do the list feel about this - how do we set a criteria for selecting
the best f/w, ID, etc for our secure networks - is it possible?

Regards

Christopher

-----------------------------------------------------------------------------
Christopher Nicholls
chrisn () dynamite com au   ~~~~~~~   chrisn () softway com au
-----------------------------------------------------------------------------
m:      0411 454755     
w:      +61 2 6243 4834 h:      +61 2 6241 2112
wf:     +61 2 6243 4848 hf:     +61 2 6241 8926
----------------------------------------------------------------------------
-