Re: Crossover Error Rate (WAS "Intrusion Prevention")

[Bennett Todd <bet () rahul net>]

Thanks for bring this up. I hadn't heard of Crossover Error Rate;
it's a nice abstraction. I'll certainly keep it in mind.

IDSes face a different problem. With sufficient [manual] tuning,
against any given sample of traffic an IDS can be adjusted to have
perfect behavior --- no false positives, nothing missed.

The problem is that the tuning needs to be done again for the next
sample of traffic.

Fortunately, successive samples of traffic taken at the same point
are similar to each other; while there is change over time, it's not
fast enough that manual tuning can't keep up with it. You can
automate incorporating frequent updates from an external source
(e.g. I pull mine automatically from www.snort.org), and
occasionally an update introduces a new false positive you have to
manually tune out.

Unfortunately, traffic taken at different locations doesn't share
this property. Each site will in general have to do some more
tuning.

Worse still, two different organizations will disagree about the
classification of a given packet or correlation. For instance, there
are some admins who think port scans are always attacks, some who
never regard them as attacks, and some who apply different standards
in different settings. One man's false positive is another man's
incident.

And for purposes of trying to have reproduceable documented measures
of performance, there's the additional problem that any such measure
is only meaningful and reproduceable against a single packet flow.
While a packet flow can be captured (e.g. tcpdump) and replayed
(e.g. tcpreplay), it will rapidly become less and less meaningful as
a measure of real world performance, since attacks (and the
signatures required to catch them) change and evolve over time.

I don't think we're in a position to make use of the Crossover Error
Rate, pretty concept though it is, because the uncontrolled
variables in the IDS world have so far defied efforts to produce
satisfying, reproduceable concrete measures of performance.

The only performance metric I've seen that I do like is "capture a
nice long slice of real traffic from the place I want to deploy the
IDS. Set up a captive LAN, use tcpreplay to send packets ripping
across the bow of the IDS. How fast can you go before you start
seeing significant packet losses?" For snort on current generation
PCI-bus machines, with quick CPUs and plenty of RAM and good drivers
for good interfaces under good OSes, the answer seems to be in the
neighborhood of 50Mbps untuned, up to as high as 200-300Mbps with
sufficiently careful tuning. Snort 2.0 is supposed to make things
better, OS developers continue to tune their stuff, and of course
hardware vendors are hard at work as well; I wouldn't be surprised
if the equivalent statement a year from now weren't 1Gbps untuned.

But an untuned IDS is typically only useful for reporting and stats
and incident investigation, if you want to try to act on alarms in
[near]-realtime you need to do some tuning. The tuning can be
minimized by careful positioning of the IDS, it's awfully easy if
you place it inside a good tight firewall, but it's still necessary,
that one man's bread problem.

-Bennett

Attachment: _bin
Description: