Re: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk

[Don Parker <dparker () bridonsecurity com>]

Well the short answer to the question of "how do you verify the IDS/IPS" is simply 
put; get it tested by someone other then the vendor. Certainly no one is calling 
any vendor a liar when it comes to their claims however I would prefer to see a 
third party attesting to a vendors claims. After all if you are going to invest a 
very large sum of money on these technologies it makes sense to invest some in  
professional 3rd party review.

Cheers,

Don

--------------------------------------------------------------
Don Parker, GCIA GCIH
Intrusion Detection & Incident Handling Specialist
Bridon Security & Training Services
http://www.bridonsecurity.com
voice: 1-613-302-2910
--------------------------------------------------------------

On Mon, 8 Nov 2004 21:01 , Paul Palmer <b.paul.palmer () gmail com> sent:

> There is another issue to consider with IPS testing that you do not
> see with IDS testing. While you can verify the detection capability of
> an IDS/IPS using these tools if the captures you use show a complete
> exploit (and not just the packets that one vendor thinks are
> interesting), it is much more difficult to verify the protection
> responses of an IPS using such tools. Detecting is a simple matter of
> verifying that a reasonable event triggers when you replay the
> capture. However, IPS devices are active devices and work by modifying
> the network traffic that flow through them in response to undesired
> events. This produces a variety of challenges.
>
> First, once the IPS responds, the remainder of the packets replayed
> are likely no longer an accurate reflection of what would have
> happened with live traffic. For example, if the IPS drops packets,
> does the capture show retransmissions at the key points? If not, how
> do you know that naturally occuring retransmissions wouldn't get
> forwarded? Does the tool compensate for this by synthesizing
> retransmissions on "dropped" packets?
>
> What if the IPS inserts TCP RST packets ahead of the undesired packet
> instead of dropping it? Does the tool account for this? How does the
> tool know whether the RST would have been effective? Does the tool
> understand the protocol stacks it is simulating to be able to answer
> that question?
>
> What if the IPS doesn't respond by dropping packets? What if it
> rewrites the attack out of the packet before forwarding it in some
> cases (because simply dropping or resetting an SMTP connection tends
> to make matters worse rather than better for example)? How does the
> tool know whether this rewriting would be effective?
>
> Ultimately, you get into a situation of role-reversal. How do you test
> these tools? Well you exercise them in "real world" scenarios to see
> how they behave. That is, you place them in a test bed with an IPS.
> You launch "attacks" and see what happens. If the test fails, you look
> to see if the tool failed or the IPS failed. Very rapidly you get into
> the situation where the IPS is used as the "test tool" for your test
> tool. That is, the IPS measures and heavily influences "proper" tool
> behavior. Tomahawk works well at testing Tippingpoint's products
> because that is how it was tested. It is horrible at testing ISS
> products (the vendor I happen to work for) because Tippingpoint has no
> vested interest in investing time and energy to make sure that ISS,
> Netscreen, McAfee, etc (I apologize to all those I left out) are
> adequately measured by their tools. I do not see how placing the tool
> in the public domain solves this problem.
>
> So who is going to invest $1,000,000 in IPS equipment from various
> vendors to properly test this tool and make sure it gives every vendor
> fair representation? Who gets to decide what is fair? The vendors? The
> public domain developers? Do you want to be one of those developers? I
> can just smell the defamation lawsuits from whichever vendors feel
> they are not being fairly represented by your code changes...
>
> FYI, I work for a vendor. We have a tool very similar to Tomahawk. It
> works well at testing our products. However, other vendor's products
> do not stand up as well to our testing :)
>
> The only reliable means of testing IPS product effectiveness I have
> discovered so far is live fire testing. Setup an isolated LAN. Place
> vulnerable systems on one side of the IPS and launch attacks from the
> other side. If the vulnerable system is compromised, the IPS failed. I
> strongly recommend modifying all of the exploits to only DoS the
> victim so that nothing more than a reboot is ever necessary to prepare
> for the next test.
>
> Paul
>
> On Sat, 6 Nov 2004 13:16:02 -0800, ADT synfinatic () gmail com> wrote:
> > (thread is getting long, so just going to snip the whole thing,
> > hopefully you kept a local copy)
> >
> > Hey Greg/Marty,
> >
> > I don't think anyone would argue that tcpreplay or tomahawk are
> > written for performance
> > testing of IDS or IPS.   I'm sure some people do that, but both have
> > rather limited use in that regards (you want to generate background
> > traffic using *your* network's traffic).  What tcpreplay and tomahawk
> > do rather well is provide the means to safely reproduce malicious
> > traffic for testing detection capabilities.
> >
> > Unlike "live tests", tcpreplay/tomahawk don't require people to
> > distribute working exploit code
> > or attack an actual host which due to the nature of exploits will
> > likely have to be "fixed" in some
> > manner.  Unlike exploit code, there is no risk that a pcap will also
> > re-format your harddrive or
> > require you to install and configure a wide variety of operating
> > systems and applications to
> > attack.
> >
> > Of course, unlike a "live test" there is some trust involved that the
> > pcap contains packets which
> > are relevant for the test you are running.   Wether or not this
> > precludes using either tool for being
> > used by someone evaluating an IDS/IPS probably depends on how much
> > they trust the pcaps.
> > For those people who don't want to trust pcaps and don't have the
> > means to get a library of working exploits, I'm sure Blade will be
> > more then happy to sell you IDS Informer (of course, now you have to
> > trust Blade, so you're just shifting your trust).
> >
> > Of course if you already have a repository of valid pcaps (maybe
> > something the OSVDB guys could do?) with known attacks, then using
> > these tools probably make a lot of sense for certain kinds of tests.
> >
> > Aaron, the tcpreplay guy
> >
> > --
> > http://synfin.net/ 
> >
> >
> >
> > --------------------------------------------------------------------------
> > Test Your IDS
> >
> > Is your IDS deployed correctly?
> > Find out quickly and easily by testing it with real-world attacks from
> > CORE IMPACT.
> > Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
> > to learn more.
> > --------------------------------------------------------------------------
>
> --------------------------------------------------------------------------
> Test Your IDS
>
> Is your IDS deployed correctly?
> Find out quickly and easily by testing it with real-world attacks from 
> CORE IMPACT.
> Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
> to learn more.
> --------------------------------------------------------------------------





--------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it with real-world attacks from 
CORE IMPACT.
Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 
to learn more.
--------------------------------------------------------------------------