Re: Open Source centralized log management/SIEM solutions

[Russell Fulton <r.fulton () AUCKLAND AC NZ>]

On 27/04/2010, at 3:02 AM, Youngquist, Jason R. wrote:

> Is anyone using any Open Source or low cost centralized log management/SIEM solution in a production environment 
> which you would recommend?
>  
> Specifically, I'm looking for:
> --scalability - must be able to handle hundreds of log sources - majority being servers and network devices
> --good searching capability
> --ability to generate alerts
> --good reporting capability – pre-built reports would be nice
> --a solution auditors would approve
> --able to meet regulatory requirements such as PCI
> --fast implementation time – how long would it take to get the solution up and running?
>  

I think getting all your logs onto a single box is the first step.  We use a linux box running syslog-ng.

Once you have that then there are various options since syslog-ng can route subsets of your logs to different 
places/applications.

We dump all of our to disk and I have a Ruby program that goes through periodically and produces alert on patterns 
using counts and pcres.  It can also filter logs to another routine for further correlation or reporting. THe most 
important thing it does is to implement what Marcus Ramum called "artificial ignorance".  THe idea is that you throw 
away all the stuff that you know and expect leaving the interesting stuff.  The catch is that it need regular 
maintenance to keep the filters up to date.  I think what is necessary to really make this work is to present the 
reports though a smart web 2.0 interface so the admin can quickly say. "hide these", "alert on those".

We trialled splunk taking a feed filtered by syslog-ng.  It handled the searching side of the operation very nicely but 
we failed to get the money to buy it.

Currently I we are looking at OSSIM (yes, that is Alienvault) and prelude but mostly from the point of view of managing 
snort data.

Another cheepish option is Aanval, I have played with it briefly again focusing more on the snort side.

The bottom line is that if you want searchability then you need to get the logs into a data base and for that to work 
well you need some very clever normalisation.  This is what splunk does really well.

For those interest in log analysis I recommend http://www.loganalysis.org/ there is a lot of information there -- you 
may even find reference to my stuff.

My experience is that the hardest of your requirements (even with money) is your last one.  Log analysis is very messy 
and very system/application dependent.

Russell

PS.  I hesitate to distribute my code now as I don't have time to develop or support it in any meaningful way :(