IDS mailing list archives
Re: Testing IDS with tcpreplay
From: Ivan Arce <ivan.arce () coresecurity com>
Date: Thu, 23 Feb 2006 18:16:55 -0300
Hi Aaron. See my comments inline as well ... Aaron Turner wrote:
Inline... On 2/20/06, Ivan Arce <ivan.arce () coresecurity com> wrote:Aaron Turner wrote:[snip]Generally speaking, tcpreplay is better when one or more of the following is true: 1) Trying to do comparative analysis and you want to make sure each device sees exactly the same thingHmm, why is that harder to accomplish with Metasploit than with tcpreplay?Because metasploit, other tools and exploits incorporate PRNGs and other methods of altering the attack so that it isn't exactly the same each time. Sure it's the same "exploit", but the bits on the wire are different. That makes "each device sees exactly the same thing"
That was exactly my point. If you are testing you IDS you'd like to know that it accurately detects an exploit for a given vulnerability not that it detects the packets that comprise a particular instance of the exploit's execution. What I tried to point out is that with tcpreplay you end up testing different things than with Metasploit or similar tools. Using tcpreplay it good to build a valid and easily repeatable set of tests but it's important to know the difference with the real thing lest you'll get burned when a different exploit for the same bug or even the same exploit in a different operational environment hits your IDS.
really difficult. Of course some tools allow you to control these via seeds but others don't.2) Need to automate or do a lot of regression testing and want a stable and relatively simple lab environmentsame as above....Again, less complex (no 2nd box and vmware to maintain/automate) and 100% repeatable data streams. Also what about attacks that Metasploit doesn't have? What if you want background traffic?
Those are certainly good points. Regarding attacks that Metasploit does not have; I guess then you'd need to find an attack for the bug elsewhere (either publicly available or in a commercial product), otherwise how would you have the replayable attack traffic anyway? As for background traffic, yep that is certainly helpful and I recognize the value of tcpreplay there, which is way smarter than simply using a traffic generator to inject random packets that are completely out of context for the test environment
3) Already have a library of pcap's (either from customers, the wild or capturing traffic of real tools like Metasploit)Yeah, but that is an entirely different kind of testing. Replaying the packets captured from the execution of an exploit is not the same as executing the same exploit again.If you're testing a vulnerable application then I agree. but if you are testing an IDS/IPS, then I would argue that it is for all intents and purposes it's the same thing. If you believe otherwise, then please explain.
Ok, I think I've touched on the point in my above comment but I will explain my view a bit more. I would argue that you are testing the IDS to figure out if it will be capable of detecting exploitation attempts against real applications in their real operational environment. If that is in fact the case, then you need to make sure that the exploit traffic you're using to test your IDS is likely the same that the IDS will see coming after the real applications. It follows then that if the exploit that generates the traffic will behave differently in the face of the real application then your captured-replayed traffic can't really assure that the IDS will work properly (unless you really understand how the exploit will behave and what traffic it will generate in any random execution). My point is that modern exploits do behave differently based on the runtime characteristics of the target system and application. I am not talking about shellcode obfuscation or NIDS evasion attempts but about exploit logic. For example if a given exploit needs to fill-up the process heap after/before triggering the bug or obtain heap layout data to change execution flow on the target app. then it will send more/less -or even different- traffic depending on the endpoint's application state.
The point being you have to put forth the effort to automate vmware/metasploit is a lot more work then just tcpreplay. Not to mention, if you want to add other tools other then metasploit now you have more automation work to do. tcpreplay is exploit/tool agnostic and provides a simple and unified method of generating all kinds of traffic- not just exploits.
agreed and understood, I am just pointing out that you will accomplish different things depending on which test strategy you choose
[snip]In general, tcpreplay isn't all that useful IMHO when you're first starting off and "want to do some IDS/IPS testing" or only intend to run a few tests or tests only once or twice unless you already happen to have a nice pcap library.Ahh that's interesting, I see it in exactly the opposite way: tcpreplay is ok when you want to scratch the surface of your IDS capabilities or perhaps more appropriate for stress or throughput testing or very basic regression testing. However, if you truly want to check if the IDS recognized real attacks you need to test with real exploit runs not with a replay of their captured traffic.Why? What is the different between "real exploit runs" vs. "replaying real exploit runs" from the viewpoint of an IDS/IPS/UTM/etc?
I think I explained this in the paragraphs above.
Perhaps you're not aware, but tcpreplay fully emulates both client and server state, from protocol setup, to the actual exploit and all the way through getting a shell/whatever if that is what you captured. An
You mean at the TCP/IP stack level or does it emulate the application's state as well?
IPS/IDS has absolutely no way of knowing if traffic was generated by tcpreplay or not, because it is EXACTLY like the traffic that was originally generated/captured. If you want to run the same exploit 500 times with different parameters with Metasploit, that's great.
You can run the same exploit 500 times with exactly the same parameters and the generated traffic will not be the same. Some examples: - The Apache-SSLv2 exploit used by the Slapper worm - An exploit for the php_memory_limit bug - An exploit for MS License Logging Service (LLSRV) The reason for this is that they have logic to take different decisions or to use different data (pointers,etc) based on the heap state of the target application.
Just capture each of those 500 attacks and replay each of them against 10 different IDS/IPS's. The point is that with tcpreplay, your test cases are 100% repeatable.
Yes, but that does not assure 100% code coverage of the exploit you used to generate the captured traffic. As I said, I see the value of using tcpreplay but the implications need to be clearly understood so you know what you're testing. -ivan --- "Buy the ticket, take the ride" -HST Ivan Arce CTO CORE SECURITY TECHNOLOGIES http://www.coresecurity.com PGP Fingerprint: C7A8 ED85 8D7B 9ADC 6836 B25D 207B E78E 2AD1 F65A ------------------------------------------------------------------------ Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 to learn more. ------------------------------------------------------------------------
Current thread:
- Re: Testing IDS with tcpreplay, (continued)
- Re: Testing IDS with tcpreplay ehanselman (Feb 14)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 15)
- Re: Testing IDS with tcpreplay Richard Bejtlich (Feb 19)
- Re: Testing IDS with tcpreplay Ivan Arce (Feb 21)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 22)
- Re: Testing IDS with tcpreplay Greg Shipley (Feb 22)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 23)
- Re: Testing IDS with tcpreplay Bob Walder (Feb 24)
- useful real-life example of IDS/IPS Shai Rubin (Feb 23)
- Re: Testing IDS with tcpreplay Stefano Zanero (Feb 26)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 15)
- Re: Testing IDS with tcpreplay Ivan Arce (Feb 23)
- IPS test machine Terry Vernon (Feb 24)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 24)
- Re: Testing IDS with tcpreplay Bob Walder (Feb 26)
- Re: Testing IDS with tcpreplay ehanselman (Feb 14)
- Re: Testing IDS with tcpreplay Bob Walder (Feb 23)
- Re: Testing IDS with tcpreplay Stefano Zanero (Feb 26)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 26)
- Re: Testing IDS with tcpreplay Aaron Turner (Feb 19)
- RE: Testing IDS with tcpreplay Bhaarath (Feb 21)