IDS mailing list archives
RE: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk
From: "Brian Smith" <bsmith () tippingpoint com>
Date: Thu, 11 Nov 2004 09:34:37 -0600
On Nov 2, Marty Roesch wrote:
As far as pcaps are concerned, pcaps in a vacuum don't really add a whole lot beyond just testing basic detection capabilities. You need to have real high grade network testing equipment like the stuff Spirent makes so that you can develop normalized, repeatable test environments in which to test detection capabilities. Measuring latency, throughput, etc is also best done in an environment where you can setup repeatable test environments or at least where you can setup repeatable baseline environments to transmit your pcaps over the top of. Tcpreplay doesn't meet this requirement particularly well all by itself, nor will the TippingPoint software.
Actually, tomahawk is less designed for testing IPS detection/blocking capabilities (although it can be used for that) and more geared toward setting up realistic, repeatable background traffic mixes. I developed it to directly address several limitations in the switch/router test gear. As you say, the good thing about the Spirent, et. al. gear is that it is very precise -- it will tell you latency within 10 ns, for example. The limitation is that the traffic generated looks nothing like the traffic that appears on a real network once you go past the headers. The same is true for most of these tools. One exception is WebAvalanche/WebReflector, which generates fairly realistic traffic, but only for a few protocols, so the mix is unrealistic. When testing a switch, router, or firewall, the traffic generated by these devices is fine. But when testing anything that goes deep into the stream, like an IPS, you need to make sure the data that it's inspecting is as realistic as possible. The traditional router/switch test gear just doesn't do that. Worse yet, they can give you misleading results because the IPS may be optimized for the traffic. As a trivial example, suppose you use smartbits to send ethernet frames padded with zeros (so that it's all zeros above layer 2). An IPS can look at that data, quickly determine that it's not IP, and send it on its way with no further processing. The test will show that the IPS has great latency and throughput, but predicts nothing about how it will perform in a real network (unless you deploy it in bypass mode :-). Tomahawk can be used to set up a repeatable traffic test using pcaps from the target network, which gives a more realistic protocol mix. The throughput stabilizes after a minute or two and is repeatable to a few percent -- some of this noise is caused by the sampling methods used to compute throughput (we sample the NIC stats, which are only updated every 2 seconds), some from the non-determinism in some IPSs (e.g., a software product using a non real-time OS will always have a tiny amount of noise, plus there's caching effects, etc.). So how do you test this stuff, given these considerations? A reasonable compromise that we use is to find the throughput limits using tomahawk with a pcap taken from the target network, and then test latency by loading the box to, say, 90% of that limit with tomahawk and use smartbits to find the latency by sending a modest stream of UDP or ICMP traffic (so the IPS can't just ignore them). Ultimately, the quality of any test is it's predictive power: that is, how well does the test predict the performance of the product in the real situation. That's how we should be measuring the effacy of these tools. If the prediction is off by an order of magnitude, those extra significant figures in the measurement don't do you a lot of good. Brian -------------------------------------------------------------------------- Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 to learn more. --------------------------------------------------------------------------
Current thread:
- RE: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk Brian Smith (Nov 12)
- <Possible follow-ups>
- RE: TippingPoint Releases Open Source Code for FirstIntrusionPrev ention Test Tool, Tomahawk Brian Smith (Nov 12)