Re: parsing logs ultra-fast inline

["Adrian Grigorof" <adi () grigorof com>]

What do we want to know?

http://www.eventid.net/firewalls/MostPopularReports.asp

The compilation of the most popular reports that we would like to see after
a firewall (or other similar device) log analysis - from a thread initiated
by mjr in the Log Analysis mailing list.

I noticed that there is a big emphasis on log parsing while there should be
more discussions about the interpretation of the log parsing results. I've
worked with logs from quite a few types of firewalls but parsing them has
never been the problem. Yes, is a tedious, frustrating job but a rather easy
one in comparison with the task of "programmatically" interpreting their
meaning. Take Tina's VPN example - how many types of log entries you would
expect from a VPN concentrator? From my experience, not more than 20 but
let's assume there are 50. Give a sample from each entry to a Perl
programmer and you will have the parsing script done in a day or two. So now
you have the data, but what are doing with it? What is relevant to a VPN
administrator? Even a seasoned security professional would appreciate some
"conclusions" that a reporting tool would provide from the data in the logs.

That being said, I agree that when you have to analyze 100 GB worth of logs,
parsing them becomes a (big) problem and you need to optimize as much as
possible. Actually, a "mere" 1 GB log is a show stopper for many analyzers
on the market.

Regards,

Adrian Grigorof
Altair Technologies
www.altairtech.ca
www.eventid.net


----- Original Message ----- 
From: "Tina Bird" <tbird () precision-guesswork com>
To: "'Marcus J. Ranum'" <mjr () ranum com>;
<firewall-wizards () honor icsalabs com>
Sent: Thursday, February 02, 2006 13:21
Subject: RE: [fw-wiz] parsing logs ultra-fast inline


marcus has been sufficiently saying what i do that i've not felt obliged to
participate in this thread, until finally:

> -----Original Message-----
> From: Marcus J. Ranum [mailto:mjr () ranum com]
> Sent: Wednesday, February 01, 2006 1:04 PM
> To: firewall-wizards () honor icsalabs com
> Subject: [fw-wiz] parsing logs ultra-fast inline

[...]

WHAT DO YOU WANT TO KNOW?

so f'r instance, imagine i've landed in a new job at a company without a
centralized logging infrastructure. the network is the usual conglomeration
of file servers, mail, web stuff, firewalls, routers, remote access. and
databases, of course. and some custom code. i'd go MAD if i tried to build
the uber-logging facility all in one go.

[...]

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards