Firewall Wizards mailing list archives
Re: RE: IDS (was: FW appliance comparison)
From: Brian Loe <knobdy () gmail com>
Date: Mon, 30 Jan 2006 09:10:03 -0600
On 1/27/06, Marcus J. Ranum <mjr () tenablesecurity com> wrote:
Joking aside, I think you got my point. It's certainly possible to handle the kind of logging load you're describing. I wouldn't go quite for far as to say it's "easy" - but your original post made it sound like you felt it was a near impossibility, or something like that.
I'll reserve opinions of possibility for when I find out how to use the data...
But you don't have to convince meIt _sounded_ like I do. After all, you were the one who said: "I'm not sure how we could do it" That's a whole different situation from if you'd said "I have to convince my management not to be stupid" in which case I would have entirely ignored your posting. After all, "I have to
I suppose, but where I was coming from was that this requires a great deal of resources - not only in space - and that requires money. So now I don't just have to convince him of the need for doing such but justify the associated costs. Naturally its slightly political, but only because I would have to put the fact that we have our pants down right now pretty delicately.
So, they obviously understand at least some of the value of collecting such information - otherwise they would have it completely turned off "for performance reasons" or something like that.
Performance is certainly not an issue, but the only use they get from it right now is the ability to fire up their desktop based viewer and watch a problem occurring real-time. That's nice and helpful for troubleshooting, to be sure, but does nothing for what you have described.
So if you're working for a company so stupid that they'd rather spend $10,000 for a "device" from Cisco (or whoever) or $100,000 for something from IBM (or whatever) then none of us can help you and your best strategy is to crawl back under a rock and go back to hoping the sky doesn't fall on your head. There's nothing any of us on this list can do to help you; you should be talking to a Cisco sales-rep, instead. But - back to your assertion that I'm saying that "they have to spend money" - wait a minute: didn't you already say you had some hardware? Presumably a computer with a hard disk, right? Maybe you don't have a terabyte in the darned thing but if you managed to do something useful and demonstrate some value to it, you'd probably find that they could afford a hard disk upgrade. But what do I know?
But wait, isn't that what I'm asking - how to go about doing just that? They've tried, I've tried, to do the logging - and always with something running on the data to help make it useful. You say it can be done and that it is useful - I'm just wanting to hear how. I think generally the mindset has been to log for troubleshooting purposes, not security (and this should probably include my last effort). This has nothing to do with stupidity, btw, but with ignorance.
So, it sounds to me like you jumped into the problem without actually thinking about it, first, and failed. That should have come as no surprise. But it appears that you generalized that failure into a theory that "it can't be done." Um, no.
I think the comment was I don't see how it could be done. I played with it as best I could before - drilling for what I thought I ought to review on a regular basis - but it didn't work. Obviously, having tried, I didn't see how it could be done. Still don't. The problem seems to get exacerbated, I might add, if you're saving all the data (no drilling) and reviewing it all manually, looking for something you've never seen before and can find no reference for.
Without a system to at least *help* you analyze it you're simply swimming in quicksand, flailing in fact.The last log analysis project I got dug into (Hi Paul!) we used really advanced tools like "grep" and "more" and "guinness stout" and figured out what the data looked like, then figured out what we wanted to do with it. Then I wrote a few carefully tailored 500 to 2000 line-of-code utilities in C that did the job, and let them run for a day and *poof* we found stuff! By the way, since I had about 40 gigs of bzipped log data that I was trying to find a single unknown event in, I had to take into account the speed of the various tools and do some back of the envelope math, first. If I'd used a database, we'd still be sitting waiting for results - and the project was 2 years ago.
If only every company could afford someone such as yourself...
As for IDS, I personally think its a mostly useless tool - especially the way they have it implemented here.But you're the guy who said: "But, on the bright side, our 2k IDS system did eventually begin blocking it from all but one customer site." comparing your "$250k" log analysis system to your $2k IDS - which certainly doesn't make it sound like you think your IDS is useless. Make up your mind, would you?
Did I say that? Can't recall the context now, but I'm thinking that I was probably discussing an already known issue that IDS eventually managed to block, after getting the appropriate signature. That's not what I would consider an IDS boast.
Remember, though, we're talking about my crunching through more than 100 times (decompressed) as much data as you were complaining about, in less time than it was taking you to collect the amount you were complaining about. That tells me your problem is probably highly solvable.
As I said, that was one firewall, we have around 8. And, as most of what I just snipped out shows, I'm at a loss for how *I* can do it better still since I'm not a developer, still have no idea what I'd be looking for...etc.,etc.. I am currently tracking things like processor and memory loads, in hopes that a jump in activity - especially at odd times - would be an alert of something bad happening. Since you mentioned it, I might see if I can better monitor numbers of log entries as well. Since your point about the data being needed after the fact I'm going to be pushing to get a place for us to store our logs - then, at the least, if something should happen we can get someone to figure it out for us. Something that seems to be lost here though is that when I was saying a Gig an hour, that was in compressed format. The logserver was rolling the logs at a gig. Not only is that a lot of data to store but that's a lot of data through the NIC. Multiplying that by two is going to be interesting, let alone 4 or more. _______________________________________________ firewall-wizards mailing list firewall-wizards () honor icsalabs com http://honor.icsalabs.com/mailman/listinfo/firewall-wizards
Current thread:
- Re: RE: IDS (was: FW appliance comparison) Marcus J. Ranum (Feb 01)
- Re: RE: IDS (was: FW appliance comparison) Brian Loe (Feb 01)
- Message not available
- Re: RE: IDS (was: FW appliance comparison) Marcus J. Ranum (Feb 01)
- Re: RE: IDS (was: FW appliance comparison) Brian Loe (Feb 02)
- RE: RE: IDS (was: FW appliance comparison) Bill Royds (Feb 02)
- RE: RE: IDS (was: FW appliance comparison) Marcus J. Ranum (Feb 02)
- RE: RE: IDS (was: FW appliance comparison) Paul Melson (Feb 02)
- RE: RE: IDS (was: FW appliance comparison) Paul Melson (Feb 02)
- Re: RE: IDS (was: FW appliance comparison) david_harris (Feb 02)
- Re: RE: IDS (was: FW appliance comparison) ArkanoiD (Feb 02)
- Message not available
- Re: RE: IDS (was: FW appliance comparison) Brian Loe (Feb 01)
- <Possible follow-ups>
- RE: RE: IDS (was: FW appliance comparison) Paul Melson (Feb 01)