[nycbug-talk] routelog -- a filter for matching, parsing and executing commands with log data

[Matthew Story]

Wanted to introduce my recently open sourced project, routelog to the talk
list:

https://github.com/axialmarket/routelog

It's a UNIX filter that matches log lines, parses them, and executes
commands using the parsed log entry ... it's basically cron, but for logs,
rather than for time.

Routelog is written in Python, using the shlex module, and implements both
a domain specific language and an interpreter. The rules file language
(routelog(5)) looks like this:

/pattern/            command

command is a shell command (just like cron, it gets passed to sh via the -c
option using execvp(2)), that takes it's positional arguments are the
results of a parsed log entry. So if you have a log line like this:

2012-12-07T12:06:11-05:00 server1 program_name: ERROR foo

$1 = 2012-12-07T12:06:11-05:00
$2 = server1
... and so on

So you can do things like:

/ERROR/      echo "$*" | mail -s "Error executing ${3%:} on $2 at $1"
error at example.com

You can also group your log arguments (again just like shell):

2012-12-07T12:06:11-05:00 server1 program_name: "these are all one arg"

The routelog(1) program emits to stdout, so you can pipline your log
processing inside a single command:

routelog mail-errors.rules /var/log/*.log | bzip2 > todays-logs.`date
+%s`.bz2

It's easily implemented by any log rotation program, via cron, or using a
queue system like fsq (github.com/axialmarket/fsq ... also available via
pip) in concert with log rotation (which is how we use it).

Any thoughts, suggestions or improvements (or ports Makefiles ... ;)) are
much appreciated.

-- 
regards,
matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20131005/892fd925/attachment.htm>