RE: On the definition of false positive - was: Re: location of an IPS

["David Goodrum" <dgoodrum () nfr com>]

So, to modify your statement a little.  A stupid little semantic really,
but...  I think you need to differentiate between false positive and false
alert.

You define false positive as an alert on something that was not actually an
attack.  My issue is with the use of the word "attack".  IDS' do not just
alert on attacks.  More and more, IDS are used to alert on network
anomalies, misconfigurations, policy enforcement, etc.

For example, we might alert you to the fact that somebody is communicating
using public community strings, or is using p2p applications, or is using a
browser version that violates your policy.   Some customers are now using us
to simply collect lists of hosts that are trying to communiate via IPv6.

So, your definitions use the word "attack", which isn't what IDS systems are
used for anymore (though it is certainly one function).

So enough preaching on semantics... here's the definitions I use:

False Positive:  something identified incorrectly.  
Example: alert on bo2k, when in fact it's simply some other application
running similar encryption.

False Alert: something identified correctly, but you don't care about it.  
Example: alert on IM activity, but you allow it.

False Negative:  something you expected us to alert on, but we missed it.

We've tried to mitigate the impact of False Alerts by assigning confidence
levels to our alerts.  So, we might alert on the fact that we see a
traceroute... but that doesn't mean we think it's an attack, so we assign it
a low confidence level.   Since this post was originally IPS focused, I'll
comment that we actually use these confidence levels to do our IPS policy.
For example, "block all attacks that NFR is 90% confident is actually an
attack".  Or "block all HTTP attacks that NFR is 90% confident is actually
an attack".  

thanks,

-dave

-----Original Message-----
From: Evil Adam Smith [mailto:eviladamsmith () yahoo com] 
Sent: Friday, October 28, 2005 1:35 AM
To: focus-ids () securityfocus com
Subject: On the definition of false positive - was: Re: location of an IPS 

Hey List,

Since Kurt obviously isn't afraid to correct others...and I know at least
one person on the list might also benefit from this comment... 

> From Kurt's post below: 
"One the one hand good, that would have been a false positive technically
speaking, otoh that's bad, it probably should have alerted on that (even if
it is a false positive)."
 
Actually, I believe it would be either a true or false negative - depending
on how you defined the terms. In this example choose to use true.
For example in the model I'm thinking of:
A false positive is when an attack is detected (positive), but it wasn't a
real attack (false) - whatever the reason the signature triggered falsely or
some such.
A true positive is when it was detected (positive) and it was a real attack
(true).
A false negative is when it wasn't detected (negative) and it wasn't a real
attack (false) - you could test for false positives with false negatives
(things the IPS shouldn't ever detect as malicious(valid traffic)).
Thus, a true negative is a real attack(true) that goes undetected
(negative).

I guess Kurt was thinking intent of the attacker matters a la an alternative
definition of "attack" but such a definition would be I believe untestable -
how would IDSes, etc. ever be able to establish the intent of a packet? If I
scan myself my ids either detected it or it did not. 

Semantic quibbles aside I don't see a more useful way to think about this
problem area using only two sets of two terms and use them in a meaningful
practical way.  

Cheers
eviladamsmith

> "Kurt Seifried" <bt () seifried org>
> 10/19/2005 09:13 PM
> Please respond to
> "Kurt Seifried" <bt () seifried org>
>
>
> To
> "Doug Fox" <dfox168 () hotmail com>,
focus-ids () securityfocus com
> cc
>
> Subject
> Re: location of an IPS
>
>
>
>
>
>
> > I'm sorry for this dumb question, which may have
been answered many
> times.
> > Where should one place an TippingPoint Unity 50
IPS device?  Behind or
> in
> > front of a firewall?
>
> Depends what you want to measure. Broadly speaking
in front of the
> firewall
> means you're measuring attempts, behind the firewall
they are penetrations
> (or do both and then compare them, that way you can
actually tell
> management
> "look we're stoping 90% of detected attacks, now
would you please let me
> tighten the firewall rules so that's 100%?" or
something). One thing to
> remember is to look for outgoing attacks as well,
that's a good indication
> of a compromised host or a hostile user.
>
> > I have a/the TippingPoint behind a Check Point
firewall. Even though we
> > externally and internally port-scanned the
firewall and the IPS many
> > times, the activity log did not contain any record
of the "attacks".
> One the one hand good, that would have been a false
positive technically
> speaking, otoh that's bad, it probably should have
alerted on that (even
> if
> it is a false positive). Sounds like you need to sit
down and do the
> setup/configuration/alerting/whatnot (aka the hard
parts of IDS/IPS).
> Broadly speaking you're saying "it's broken" to
which I can only say
> "bummer. try fixing it."
>
> > What am I missing here?  Any pointers are
appreciated.
> > Thanks,
>
> The dreaded C word comes to mind (consultant), if
your company lacks the
> expertise to set this up buy someones time who does.
>
> -Kurt


                
__________________________________
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from CORE
IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
to learn more.
------------------------------------------------------------------------



------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------