Re: Testing IDS with tcpreplay

[Stefano Zanero <s.zanero () securenetwork it>]

Greg Shipley wrote:

> 1. Philosophically, as a tester I believe it is important to decide
> whether you want to validate that a device is detecting actual attacks
> vs. detecting a specific set of packets.  

It's not a matter of philosopy, actually. It's the core of a testing
process - deciding what you want to test !

> and sometimes those packet captures (used for the replay process) are
> flawed themselves. 

Big underline, yellow marker, and large "!!!!!" on the margin.

Signatures that are so brittle that they react differently to different
runs of the exploits are not worthy to be counted as positives :)

> think I've saved DAYS of my life by preemptively ending finger pointing
> thanks to mr. shell

ROTFL. I will reuse this phrase, unless it's copyrighted: it expresses
my own feelings very well :)

> 6. I think MetaSploit, CORE's product, and ImmunitySec's Canvas are good
> foundations for executing real attacks.  

Agree. Not the only ones, but good starting points.

One thing that IMPACT and MetaSploit miss is the ability of adding in
some evasion techniques, which Canvas can do. On the other hand, you can
add e.g. fragmentation to Metasploit (and I suppose IMPACT, didn't try)
with the usual tools, so it may not be a decisive advantage for Canvas.

I liked the Canvas Reference Implementation initiative. Hope to see more
of the same coming from Immunity.

> However, without sufficient LEGIT background traffic (see Spirent's
> WebAvalanche for more info here) 

I strongly disagree that HTTP can be thought as "good" background
traffic, but if you reformulate it as "testing an HTTP-based attack
emulating a mostly-HTTP environment such as a webfarm DMZ" this can hold.

However, I'd like to stress (in particular for IDS probes that are
supposed to be placed into LANs) that HTTP traffic, IN PARTICULAR if
generated by load generation tools such as Spirent's, is not, by any
standard, a close match to real-world condition.

In a LAN you have mail traffic, SMB traffic, instant messaging, and (God
save us) peer to peer and even GAMING traffic going through from time to
time. The capability of an IDS to properly detect attacks on different
protocols in a mix of ever-changing traffic such as this is an important
feature to test.

>  This is a much bigger subject, but the short
> version is NO BACKGROUND TRAFFIC = TEST IS LIMITED IN VALUE.

Of course !

> 8. Driving this point one step further, using replay to put attack
> patterns on the wire is one thing, using it to put background traffic on
> the wire is another thing

I'm sure you meant what I mean, but let me explicit one thing on this
point. If you mix attack traffic over background traffic which was
generated in a different context (say, different network, use of load
generators, etc), you introduce subtle differences. For instance, IP
numbering, TTLs of packets, TOS... The DARPA dataset is a good example
of how well-meaning tests can go bad.

This may not matter if you use a purely pattern-matching engine, but if
your device is context-aware or has anomaly detection capabilities, then
it could easily spoil your tests by flagging as anomalous your attack
packets because they belong to strange source/destination couples, or
because they have strange TTL or TOS fields, etc.

I do not exhaustively know which commercial products can do this, but
surely some devices from Arbor and Lancope can.

So, mixing traffic is BAD, co-generating attacks and background on a
testbed is the way to go.

> Ok, final point: if you don't have a
> well equipped lab the ideal scenario for many orgs is piloting a device
> using their own internal traffic, and then using controlled attack and
> victim boxes.

I totally agree :)

Bye,
Stefano

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------