nanog mailing list archives
Re: improving signal to noise ratio from centralized network syslogs
From: Brian Knight <ml () knight-networks com>
Date: Mon, 05 Feb 2018 17:43:02 -0600
On 2018-02-03 15:49, Scott Weeks wrote:
Then, you can watch your network in real time like so (below is all one line): tail -f /var/log/router.log /var/log/switch.log | egrep -vi 'term1|term2|termN' 'egrep -v' takes out all the lines you don't want to see while the syslog messages scroll across the screen.
Syslog-ng can do regex filtering on messages also. So instead of doing an 'egrep -v' on a huge file after it has been logged, you can put your filter right into the syslog-ng configuration, and have those filtered messages output to a file (or any other output that syslog-ng supports). The result is a smaller file to search and work with.
We implemented a simple email alerter using this functionality. In syslog-ng, we set up two filters. One filter does the 'egrep -v':
filter f_email_msg {not message("%PKT_INFRA-LINEPROTO-.*[0-9/]+\\.") # filter out subinterface up/downs
and not message("%PKT_INFRA-LINEPROTO-.*Multilink") and not message("%PKT_INFRA-LINEPROTO-.*Serial") and not message("%PKT_INFRA-LINEPROTO-.*Tunnel") # etc };Another filter applied to the messages filters messages to just our core devices:
filter f_email_sources { host("192.0.2.1") or host("192.0.2.2") or host("192.0.2.3") or host("192.0.2.4") or host("192.0.2.5") or host("192.0.2.6") }; Then those are tied together in a syslog-ng rule that outputs to a file: destination d_email_log { file("/var/log/syslog-ng/alert/alerts.log" template("$HOST:$MSG\n") create_dirs(yes) ); };log { source(s_devices); filter(f_email_sources); filter(f_email_msg); destination(d_email_log); };
A lightweight Python script that runs as a daemon checks that file once every 10 seconds, and if the file length is non-zero, it sends the contents of the file in an email to the admins. A shell script run as a cron job would work equally as well.
(Also, for emailed syslogs, there is more incentive for the admin to keep her or his message filter up to date, as opposed to a file the administrator must manually examine. Otherwise the admin has a full inbox :) )
It's very simple and stable, and has worked better than the commercial product we used to use for this purpose.
-Brian
Current thread:
- Re: improving signal to noise ratio from centralized network syslogs Scott Weeks (Feb 03)
- Re: improving signal to noise ratio from centralized network syslogs Tarko Tikan (Feb 04)
- Re: improving signal to noise ratio from centralized network syslogs Shane Short (Feb 04)
- Re: improving signal to noise ratio from centralized network syslogs Brian Knight (Feb 05)
- <Possible follow-ups>
- Re: improving signal to noise ratio from centralized network syslogs Scott Weeks (Feb 05)
- Re: improving signal to noise ratio from centralized network syslogs valdis . kletnieks (Feb 05)
- Re: improving signal to noise ratio from centralized network syslogs James Bensley (Feb 05)
- Re: improving signal to noise ratio from centralized network syslogs valdis . kletnieks (Feb 05)
- Re: improving signal to noise ratio from centralized network syslogs John Kougoulos (Feb 06)
- Re: improving signal to noise ratio from centralized network syslogs valdis . kletnieks (Feb 05)
- Re: improving signal to noise ratio from centralized network syslogs Scott Weeks (Feb 05)