Snort mailing list archives
Re: Anyones doomsday machine running low on IDS analyst tears?
From: Will Metcalf <william.metcalf () gmail com>
Date: Thu, 7 Oct 2010 05:19:12 -0500
Steve, Thanks for the additional info. It was more of a heads up for folks switching to a VRT snort/2.9.0 conf to review their existing rule-sets. If people have custom rules updating to this version of the conf can have a significantly negative performance impact for some sigs. Of course if you don't update to include other portions of the VRT/snort-2.9.0 config like extended_response_inspection you don't have access to file_data so VRT sigs that rely on this won't fire. Regards, Will On Wed, Oct 6, 2010 at 10:01 PM, Steven Sturges <steve.sturges () sourcefire com> wrote:
The max_pattern_len is a memory option that is used in conjunction with ac-split (introduced in 2.8.6). In terms of the pattern matcher itself, it gets near AC-FULL performance at the AC-BNFA memory. Pretty significant when there are 10k+ rules in your configuration. Performance-wise, reducing the size of the pattern matcher is two-fold. -- Use of the faster AC algorithm -- Fewer CPU cache misses during pattern matching phase The whole idea of the fast pattern matcher is to use the most unique patterns to select a set of "most likely" rules for further evaluation. One could argue that of the pattern "User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)", the real unique part starts with "Mozilla", so the pattern itself could be shortened. Every HTTP request has a User-Agent header these days, so that really isn't buying you all that much in terms of uniqueness of the pattern itself. And when it comes to evaluation of the content, that's an extra 12 characters that need to be compared via boyer-moore. In the specific rule example, if "User-Agent:" is a required part of the pattern, using an offset of 13, length 20 would get you "Mozilla/3.0 (compati" -- eg: content:"User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)"; fast_pattern:13,20; Pretty unique & short. Therefore less memory in pattern matcher -- that can now be used for Stream reassembly, gzip decompression, etc. Will's solution in the follow-on post of fast_pattern:30,20 also works and gives you the tail of the above pattern. Or, split the two contents: content:"User-Agent|3A| "; http_header; content:"Mozilla/3.0 (compatible|3b| Indy Library)"; distance:0; fast_pattern; Or, increase the max pattern length in the default config. Rule writers know the most unique part of the content patterns they are using, especially as they relate to patterns in other rules, and the parameters to fast_pattern give them the tools that are needed. Snort is highly configurable and users have so many options to work with here to suit their specific ruleset, sensor, available memory, network speed, etc. Or, you can simply revert to using the slower AC-BNFA configuration of yore. Cheers. -steveFrom: *Will Metcalf* <william.metcalf () gmail com> Date: Wed, Oct 6, 2010 at 3:10 PM To: Emerging Threats Signatures <emerging-sigs () emergingthreats net> Cc: Snort Users <snort-users () lists sourceforge net> I kid... I kid.. But seriously ran into something sort of interesting playing snort conf options. It seems that the default pm in snort has changed the the default pattern matcher to be ac-split via... # Configure the detection engine See the Snort Manual, Configuring Snort - Includes - Config config detection: search-method ac-split search-optimize max-pattern-len 20 If you are using your own custom rule sets/ET rules this means that if you previously had an override for fast_pattern's default cut-off of 20 via offset and length options via fast_pattern:<offset>,<length>; or you expected the entire unique pattern specified via fast_pattern, You are now limited to only 20 bytes for fast_pattern even if you set offset/length. So for example let us consider the following ET signature optimized for 2.8.6 using the default pm in 2.8.6 of ac-bnfa. alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET TROJAN Delf Checkin via HTTP (8)"; flow:established,to_server; content:"POST"; http_method; content:".php"; http_uri; nocase; content:"User-Agent|3a| Mozilla/3.0 (compatible|3b| Indy Library)"; http_header; fast_pattern; content:"name="; http_client_body; depth:5; classtype:trojan-activity; reference:url,doc.emergingthreats.net/2008268; reference:url, www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Delf; sid:2008268; rev:5;) Fast pattern matcher: HTTP Header content Fast pattern set: yes Fast pattern only: no Negated: no Pattern offset,length: none Pattern truncated: no Original pattern "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)" Final pattern "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)" timestamp: 1286391049 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 18 4 4 55 3.1 3.8 2.9 timestamp: 1286391054 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 18 4 4 52 2.9 3.8 2.7 timestamp: 1286391058 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 18 4 4 54 3.0 3.9 2.8 Ok this is what I expect. I have a long match that should be fairly unique. Lets modify the pm to be the one included in the VRT version of the default snort.conf and/or the 2.9.0 snort.conf. We now get "User-Agent:|20|Mozilla/" as the match added to fast_pattern. Uh oh... This isn't a very unique pattern is it, this will cause all packets that contain a firefox UA to be at least partially evaluated. 1:2008268 Fast pattern matcher: HTTP Header content Fast pattern set: yes Fast pattern only: no Negated: no Pattern offset,length: none Pattern truncated: 50 to 20 bytes Original pattern "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)" Final pattern "User-Agent:|20|Mozilla/" timestamp: 1286390771 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 17381 4 4 10212 0.6 4.7 0.6 timestamp: 1286390775 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 17381 4 4 9516 0.5 4.7 0.5 timestamp: 1286390779 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 17381 4 4 9347 0.5 4.5 0.5 timestamp: 1286390784 Rule Profile Statistics (all rules) ========================================================== Num SID GID Rev Checks Matches Alerts Microsecs Avg/Check Avg/Match Avg/Nonmatch === === === === ====== ======= ====== ========= ========= ========= ============ 1 2008268 1 5 17381 4 4 9266 0.5 4.1 0.5 Evaluating the same pcap we now spend 20x more ticks evaluating the same rule. I doubt that specifying a fast pattern offset and length will allow us to override the max-pattern-len 20 but lets give it a shot by specifying 0,50 as the offset and length. snaplen = 65535 1:2008268 Fast pattern matcher: HTTP Header content Fast pattern set: yes Fast pattern only: no Negated: no Pattern <offset,length>: 0,50 "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)" Pattern truncated: 50 to 20 bytes Original pattern "User-Agent:|20|Mozilla/3.0|20|(compatible|3B 20|Indy|20|Library)" Final pattern "User-Agent:|20|Mozilla/" No dice.. So I guess the take away here is that if you are moving to a VRT snort.conf or a 2.9.0 ruleset and you are running custom rules I would pay real close attention to debug-print-fast-pattern output. We are going through the poor performers now and making modifications where appropriate for ET rules, just thought folks might want to know ;-)... Regards, Will
------------------------------------------------------------------------------ Beautiful is writing same markup. Internet Explorer 9 supports standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3. Spend less time writing and rewriting code and more time creating great experiences on the web. Be a part of the beta today. http://p.sf.net/sfu/beautyoftheweb _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users
Current thread:
- Anyones doomsday machine running low on IDS analyst tears? Will Metcalf (Oct 06)
- Re: Anyones doomsday machine running low on IDS analyst tears? Will Metcalf (Oct 06)
- Message not available
- Re: Anyones doomsday machine running low on IDS analyst tears? Steven Sturges (Oct 06)
- Re: Anyones doomsday machine running low on IDS analyst tears? Will Metcalf (Oct 07)
- Message not available
- Re: Anyones doomsday machine running low on IDS analyst tears? Will Metcalf (Oct 06)