Snort mailing list archives
Re: HELP ON SNORT
From: Martin Holste <mcholste () gmail com>
Date: Mon, 30 Jan 2012 10:09:50 -0600
@martin How are you building time based metrics on 1 million records in less then a second? This all sounds pretty intense.. i'll pull the source and give it a go.
Just run "contrib/install.sh node" and "install.sh web", and it should pretty much install itself. A VM of Ubuntu works great. MySQL is only used for storage, not searching. All queries hit Sphinx (sphinxsearch.com, the engine that powers Craigslist). Sphinx does pseudo-map-reduce and full-text searching, and is orders of magnitude faster at indexing than the public version of Google's BigTable (Hbase). Now that PostgresQL 9.1 has non-doublewrite-buffered tables like MySQL, you could use it instead if you needed to. The gist is that MySQL can batch load at 100k rows/sec, and Sphinx can index at 100k rows/sec when reading from MySQL. So, if you run 1-minute batch jobs, your peak rate is 100k events/sec, with a sustained rate of 30k/sec (due to occasional consolidation which is necessary). This is pretty technical, but I think it's a good thing to point out on the list because Dustin's point is dead-on: you can't expect any standard database to perform advanced analytics at high volumes (unless you're throwing a massive amount of hardware at it, which is out-of-scope for this discussion). So, either you tune your sensors so the volume is low, or you use something log-based, like ELSA or Splunk. Also, contact me off-list or on the ELSA list at http://groups.google.com/group/enterprise-log-search-and-archive if you have any questions on it. ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users Please visit http://blog.snort.org to stay current on all the latest Snort news!
Current thread:
- Re: HELP ON SNORT, (continued)
- Re: HELP ON SNORT Jefferson, Shawn (Jan 30)
- Re: HELP ON SNORT Lay, James (Jan 30)
- Re: HELP ON SNORT Jeremy Hoel (Jan 30)
- Re: HELP ON SNORT Dustin Webber (Jan 30)
- Re: HELP ON SNORT beenph (Jan 29)
- Re: HELP ON SNORT Dustin Webber (Jan 30)
- Re: HELP ON SNORT beenph (Jan 30)
- Re: HELP ON SNORT Martin Holste (Jan 30)
- Re: HELP ON SNORT Dustin Webber (Jan 30)
- Re: HELP ON SNORT beenph (Jan 30)
- Re: HELP ON SNORT Martin Holste (Jan 30)
- Re: HELP ON SNORT Dustin Webber (Jan 30)
- Re: HELP ON SNORT Carney, Megan (Jan 30)
- Re: HELP ON SNORT Rich Graves (Jan 31)
- Re: HELP ON SNORT Jeremy Hoel (Jan 29)
- Re: HELP ON SNORT Scott Runnels (Jan 29)
- Re: HELP ON SNORT Jeremy Hoel (Jan 29)
- Re: HELP ON SNORT Heine Lysemose (Jan 29)
- Re: HELP ON SNORT Eric G (Jan 31)
- Re: HELP ON SNORT Kimi Ushida (Jan 30)