IDS mailing list archives
RE: Random IDS Thoughts [WAS: Re: IDS thoughts]
From: "Steven Rudolph" <srudolph () iocenter net>
Date: Tue, 10 Jun 2003 08:43:04 -0400
Mike, Thank you for sharing this with everyone. You had mentioned that you have home grown your log collection system. If you are using any open source programs to do this, what have your choices been? I am attempting to build what sounds like a similar setup but on a much smaller scale - about 500 servers or so with sustained bandwidth in only the 10Mb range out to the net. I am still in the development/proof of concept stage and experimenting with different ideas at the moment. I would like to consolidate logs from syslog (using msyslog), Windows (syslogNT), and application logs. I am just starting the hunt for application log -> SQL database import utilities for both Apache, IIS and some others. Could you recommend any programs that are capable of doing this? Could you point me towards some papers or web sites that overview data mining techniques? Thanks, Steve Rudolph, CCSA, CCSE Internet Operations Center -----Original Message----- From: Mike Lyman [mailto:mlyman () west-point org] Sent: Saturday, June 07, 2003 1:52 PM To: focus-ids () securityfocus com Subject: RE: Random IDS Thoughts [WAS: Re: IDS thoughts]
Hint: data mining techniques, anyone ? There's a great book by J. Mena on the topic, which I warmly recommend.
I don't think I've posted here before so to set this up, I've been running and building the IDS systems on a global network for about three to four years. 60,000+ employees and contingent staff, 300,000+ systems on the network and Internet egress and ingress in over two dozen locations around the world. Data overload is an understatement for what we face. The value of data mining on IDS data was first demonstrated to us by folks in our research group who had wanted to do a project on our IDS pilot data. They showed us stuff we'd have never seen even with today's consoles on the commercial IDS systems we use. Since that time we have more and more mining the data and twisting it this way and that. The single most common skill we put on job requirements is the ability to run SQL queries and that is a high priority on our training schedules. Through developing differenent views of all the data available to us and constant analysis, we've been able to create reliable alerts with few false positives from our commercial systems. With home grown log collection, we've been able to craft low noise, high signal alerting IDS systems from normal high noise event logging. All of it is finely tuned for our environment instead of generic enviroments that the IDS venders have to try to shoot for. It is no where near pefected yet but it is far more managable that what we used to have even though we now have considerably more data sources. If you are not looking into data mining techniques, you are missing a great way to use your data and reducing the data overload. Mike Lyman CISSP mlyman () west-point org pgp keyid 0xD7BBADAD ------------------------------------------------------------------------ ------- INTRUSION PREVENTION: READY FOR PRIME TIME? IntruShield now offers unprecedented Intrusion IntelligenceTM capabilities - including intrusion identification, relevancy, direction, impact and analysis - enabling a path to prevention. Download the latest white paper "Intrusion Prevention: Myths, Challenges, and Requirements" at: http://www.securityfocus.com/IntruVert-focus-ids2 ------------------------------------------------------------------------ -------
Attachment:
smime.p7s
Description:
Current thread:
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] Stefano Zanero (Jun 02)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Mike Lyman (Jun 07)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Roger A. Grimes (Jun 07)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Mike Lyman (Jun 07)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Roger A. Grimes (Jun 07)
- <Possible follow-ups>
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] Stefano Zanero (Jun 02)
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] Magnus Almgren (Jun 03)
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] SecurIT Informatique Inc. (Jun 03)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Steven Rudolph (Jun 12)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Mike Lyman (Jun 13)
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] Bill Royds (Jun 13)
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] oudot laurent (Jun 17)
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] Devdas Bhagat (Jun 14)
- Re: Random IDS Thoughts [WAS: Re: IDS thoughts] Anton A. Chuvakin (Jun 17)
- RE: Random IDS Thoughts [WAS: Re: IDS thoughts] Mike Lyman (Jun 07)