Re: Snort and high performance networks

["snort user" <snortuser () hotmail com>]

Chris,

Sorry I think may have confused people by  using  double negative when I 
said 800 Mb/s is not impossible. I agree 800 Mb/s is totally possible. It 
just requires dealing with the interrupts and a lot of fine tuning . I have 
looked into this and I an using a libpcap-mmap version with a buffer at the 
NIC. Also the kernel has been modified so that it will only get interuppted 
once and then packets will fill the buffer until polled. Once the polling 
has happened interupts are again enabled for the NIC.

BUT, when all rules and preprocessors are enabled snort cannot handle this. 
The main bottle neck actually appears to be in snort itself . I have even 
gone so far as to set the affinity (possible now in the 2.6.x kernel) for 
snort to its own processor. It still pegs 100% of the processor and this is 
on a lightning fast P4. Now if I drastically cut down the rules and remove 
some preprocessors or allocate them enoumous amounts of memory I can get 
close to 800 Mb/s .

SCAMPI was able to get snort to run in the mulit-gigE but take a look at 
what this required by a large team of PHD networking and security 
professionals to achevieve this. They even say snort on a P4 cannot handle 
more than 350 Mb/s in a somewhat default configuration.   And this was a 
year and a half project.
http://www.ist-scampi.org/publications/papers/sourdis-fccm2004.pdf

Three research states  the bottle neck is the pattern matching in snort, and 
this is the reason it cannot handle the high speeds.

Id still like to hear if anyone is actaully acheiving gigE speeds? And can 
actuallly state what technology and mods youd did you achieve this.

-- UoC --




> To be fair, I think that 800Mbps is completely and totally possible. I  can 
> capture 400mbps on a  dual P3 800  - I'm not running it through  snort but 
> my own set of software which might be more computationally  intensive. Now, 
> my system isn't set up all that well - if I was smart I  would have gotten 
> a single proc box - the dual system just eats up too  many interrupts - 
> which is where my main problem is. The syskonnect  card just throws off too 
> many interrupts for the hardware to keep up.  With some modifications 
> including more aggressive interrupt coalescing,  wedging a larger buffer 
> into the libpcap code, and possibly some minor  kernel mods I can see it 
> hitting 800Mbps. Even better would be getting  it into a more up to date 
> box with a single proc and PCI-X.
>
> My specific problem is that we're - essentially - an all optical house  
> here so I need to use two GigE cards to capture both inbound and  outbound 
> - which means more hassles with interrupts and so forth and  such not. Not 
> a good time.
>
> Anyway, I'll be talking with someone who used to be with Enadace and  the 
> DAG project over at Waikato (en zed university). I saw some of the  work 
> they've been doing  at the PAM 2001 conf and have been keeping an  eye on 
> them since. After the presentation I'll sit down and see what he  has to 
> say about this issue and post some highlights here.
>
>
> On May 21, 2004, at 11:58 AM, snort user wrote:
>
> > Hi,
> >
> > Ive snipped out some of the recent posts to this thread. Weve been  doing 
> > extensive research into snort speeds at my University and to me  it seems 
> > like these 2 posts are completely innaccurate and absurb.  Chad claims to 
> > capture all traffic with all rules and preprocessors  with a $2500 piece 
> > of hardware, while if you buy a $50,000 solution  from Sourcefire(home of 
> > the creator of snort) you can only get 1 Gig  and they disable rules and 
> > preprocessors  
> > (http://osec.neohapsis.com/results/nids/sourcefire-ns3020f-2.6 
> > -06.25.2003/productinfo.html).
> > And then when Chris asked you your specs on your box you differ him to  
> > TopLayer.
> >
> > Even getting 800 Mb/s as Rafael said is not impossible but  really is  not 
> > feasible without hardcore kernel modifcation and maybe even  silicon chips 
> > and ASIC cards.
> >
> > Would either of you like to share how your able to do this, I mean the  
> > technologies and hardware you using?  Also how do you verify these  
> > results?
> >
> > -- UoC --
> >
> >
> > -- snip Rafael Ortega--
> > > I'm currently snorting close to 800Mbps with no problem.  What to do  
> > > with
> > > the amount of info, is another story.  I tried ACID, but after 24  hours 
> > > and
> > > 700,000 events registered, the data base becomes too slow, even after
> > > indexing certain reference fields.
> > -- end snip --
> >
> > -- snip Kreimendahl, Chad --
> > > FWIW... I've got systems that are easily handling between 3-4Gbps  each.
> > > That's partially hardware, partially OS, and a little tiny config  work.
> > > Very near to all rules enabled on these interfaces, as well as all of
> > > the preprocessors (minus the broken ones), and a database output  plugin.
> > >
> > > 0 dropped packets.   If you check the archives for this list, you'll
> > > find discussions about kernels that can do polling against network
> > > devices, and how this enhances snort performance on high speed links
> > > (network performance in general, really).  I believe I mention the  OSes,
> > > maybe some config info and hardware used.
> > -- end snip --
> >
> > _________________________________________________________________
> > FREE pop-up blocking with the new MSN Toolbar – get it now!  
> > http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: Oracle 10g
> > Get certified on the hottest thing ever to hit the market... Oracle  10g. 
> > Take an Oracle 10g class now, and we'll give you the exam FREE.
> > http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> > _______________________________________________
> > Snort-users mailing list
> > Snort-users () lists sourceforge net
> > Go to this URL to change user options or unsubscribe:
> > https://lists.sourceforge.net/lists/listinfo/snort-users
> > Snort-users list archive:
> > http://www.geocrawler.com/redir-sf.php3?list=snort-users

_________________________________________________________________
Stop worrying about overloading your inbox - get MSN Hotmail Extra Storage! 
http://join.msn.click-url.com/go/onm00200362ave/direct/01/



-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users