Re: Testing IDS with tcpreplay

[Bob Walder <bwalder () spamcop net>]

Again - I have to agree - GIGO is a huge problem but that is no reason to
reject replay tools out of hand. Greg mentioned that some testers fail to
break out the Ethereal - well those testers should be avoided!

We run the exploits live and verify the result. We capture that traffic,
including traffic from any root/admin shell we get. We vet the PCAP
carefully. We then run the PCAP through the replay tool and capture THAT
traffic, checking for typical problems such as invalid checksums, TTL values
which have been changed, etc, etc. If that traffic is no good we don't run
that particular exploit through a reply tool, we only run it live.

If we get an unexpected "miss" in the lab, we are always ready to run the
live exploit. And we do still run a good proportion of live exploits anyway,
when we are aware that they cause problems with some replay tools. In other
words, the replay tool is just ONE in the armoury for the serious tester.

However, for the casual tester (i.e. The guy who just installed Snort and
wants to make sure it is detecting something) then the replay tool with a
GOOD set of PCAPs is all he really needs - no need to bother with live
exploits at all. Then every time he gets a software update or makes config
changes, he can simply replay his PCAPs and make sure everything is still OK

Horse for courses IMHO - to say "replay tools bad, live exploits good" all
the time is a bit extreme.

Bob


On 23/2/06 08:40, "Aaron Turner" <synfinatic () gmail com> wrote:

> Hey Greg,
>
> I think you make some good points.  If I could dare to offer to
> summarize your argument against replay tools it would be "garbage in,
> garbage out".  And it's something I'd have to agree with 100%.  If
> you're not willing to take the time to make sure your captures contain
> the "correct" information (however that might be defined) then you're
> asking for trouble.  It's one of the reasons why I haven't tried
> making pcaps available for public consumption.
>
> I hope nobody thinks tcpreplay/tomahawk/IDS Informer/TrafficIQ are the
> best solution for the entire IDS/IPS testing space, because they're
> clearly not.  There are some areas (like regression) where it tends to
> work well (at least certain vendors have told me so) and others where
> it falls flat.  It may not be 100% accurate, but doing valid tests 90%
> of the time is better then no tests 100% of the time.
>
> IMHO, the difference between "actual attacks" and "specific sequence
> of packets" is that you haven't verified that your sequence of packets
> is the correct representation of the actual attack.  In a controlled
> lab environment, it's not hard, but it takes effort and people like
> shortcuts.  (Note to those still reading: If you're not including that
> shell, reverse socket or whatever in your pcap showing the attack was
> sucessful, you're leaving out important information.)
>
> Luckly for everyone there are plenty of free and commercial tools out
> there to fill your toolbox with.  I encourage everyone to do their
> homework and figure out how to base-line their tools... after all
> these tools contain code developed by humans and probably have as many
> bugs as the devices they're testing.  If you ever look at the
> tcpreplay changelog you'll know what I'm talking about. :)
>
> And now to summarize my email: YMMV.
>
> Regards,
> Aaron
>
> --
> Aaron Turner
> http://synfin.net/
>
> ------------------------------------------------------------------------
> Test Your IDS
>
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it
> with real-world attacks from CORE IMPACT.
> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708
> to learn more.
> ------------------------------------------------------------------------



------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
------------------------------------------------------------------------