Snort mailing list archives
Re: Announce: FLoP-1.0 --- Fast Logging Project for snort (fwd)
From: Dirk Geschke <Dirk_Geschke () genua de>
Date: Tue, 02 Dec 2003 14:31:37 +0100
Hi Bamm,
I've learned to expect the unexpected. For me, many of my sensors are remote, and I don't control any of the networks. So, if we take our 'central' network down for maintenance, my remote sensors wouldn't be able to connect to the central servsock until it's back up. Usually this is no more than two to four hours, but every once in a while it can go down for longer. Yes, we write alerts to disk after they use X amount of mem (set in a conf file). They are written one alert per file and then read back in order once the backend becomes available again. There are numerous ways to accomplish this and I like the extra reliability it provides. I don't think it's right to compare it to barnyard since we are only writing to disk when we can't send the alerts up. I'd rather take a small hit in my snort performance rather than losing alerts altogether.
the idea of FLoP is to have a fast logging. Therefore I assumed a dedicated network for the communication between the remote sensors and the central server. So in addition there neither encryption of the data nor is there any validation of the sensor (aka sockserv) connecting to the central server. (On stealth sensors this is just a need to have a second network. But indeed I think this network should be resevered for the NIDS.) Further if you don't get too much alerts then sockserv should still be able to work (except for power shortage). The problem is if you get too many alerts. But then I see the problem of re-inserting the stored files in the same way. When will you do this if the actual attack rate is high. Do you then want to store the actual alerts on the disk and first insert the old ones?
Hmm, this is a little bit more complicated. I don't know how I can identify problems with the database connection within servsock. But the good thing is: Good databases will never die ;-)We can dream can't we ;)
Oh yes, at least sometimes... But indeed I never had problems with abnormal dieing of databases. (A full disk is an abnormal situation.)
In fact databases will rarely die and my project assumes that the database is running on the same host as servsock. Otherwise we can't feed the database via an unix socket... So a reboot is no issue. DB cleaning could be: but this can be done online. Then maybe servsock will work a little bit slower...Depends on the database. We use postgres and while a 'normal' vacuum is good for routine maintenance, once you postgres DB gets huge (tens of millions of rows) you probably need to run 'VACUUM FULL' which will totally lock the DB until its done (were talking hours here). I'd be interested to see how servsock respondes to a locked DB. Generally, I restart the DB and only allow local connections when I need to do a 'VACUUM FULL', this causes the remote agents to start buffering their events locally (our version of servsock isn't run on the DB machine).
Hmm, I fear servsock will run into trouble if the database id off. Before tearing down the database you can stop servsock so no new alerts are accepted, the old ones are stored and then you can wait for the database to come online again. Then restart servsock. If the parameters for sockserv are chosen appropiate all should go on as before. That's at least a possibility. Stopping the database during servsock is running will simply drop the alerts to nirwana. (Ok, you will get an error message from the client library wich is printed to stdout or syslog which contains the failed statement.) But it is no good idea at all. But how many sensors are you running to get such amount of data even if the connection to the sensors go off for a few hours inbetween? I think a database should be cleare by some automatically process to spool off old or less important data. Who really cares about alerts of the last month? Yes, you should think of making a backup but I think it is not really required to be part of a vital database. [...]
Which database design are you using? I think it is the same one as of snort/ACID? Did you ever think about the possibility to store the whole pcap data in the database? This should make it unnecessary to store them on disk on a separate way.No, sguil does not use the standard snort/acid schema. There have been many discussions about this, but basically the standard schema isn't as scalable as I and others need/want it to be. I can send you a diagram (once I create it) otherwise you can just take a peek at the create_sguildb.sql script in the source. Pcap data for each alert is stored in the DB. The pcap that is stored on the sensors is for entire streams (think 'log ip any any -> any any'). It's hard enought to store that data locally, I can't imagine trying to push that data up to a DB (let alone dealing with it once it's there). Sguil uses barnyard and the op_sguil plugin for recieving RT events and INSERTing them into the DB right now. Are you interested in discussing (off this list) an option for FLoP to use the sguil schema and to send alerts directly to sguild?
I just took a short look at create_sguildb.sql. At first glance it does not look very different in contrast to the ACID database but contains some fields which are not included in the FLoP "design". Is it possible that the tables contain a lot of redundancy? This seems not to be very fast... But I have to look at in a silent moment and read the documentation of sguil too. But this will take some time which I don't have in the moment... Best regards Dirk ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users
Current thread:
- Re: Announce: FLoP-1.0 --- Fast Logging Project for snort (fwd) Dirk Geschke (Dec 02)