[nycbug-talk] routelog -- a filter for matching, parsing and executing commands with log data

[Okan Demirmen]

Check out sec as well: http://www.estpak.ee/~risto/sec/

On Sat, Oct 5, 2013 at 7:27 PM, Matthew Story <matthewstory at gmail.com> wrote:
> Wanted to introduce my recently open sourced project, routelog to the talk
> list:
>
> https://github.com/axialmarket/routelog
>
> It's a UNIX filter that matches log lines, parses them, and executes
> commands using the parsed log entry ... it's basically cron, but for logs,
> rather than for time.
>
> Routelog is written in Python, using the shlex module, and implements both a
> domain specific language and an interpreter. The rules file language
> (routelog(5)) looks like this:
>
> /pattern/            command
>
> command is a shell command (just like cron, it gets passed to sh via the -c
> option using execvp(2)), that takes it's positional arguments are the
> results of a parsed log entry. So if you have a log line like this:
>
> 2012-12-07T12:06:11-05:00 server1 program_name: ERROR foo
>
> $1 = 2012-12-07T12:06:11-05:00
> $2 = server1
> ... and so on
>
> So you can do things like:
>
> /ERROR/      echo "$*" | mail -s "Error executing ${3%:} on $2 at $1"
> error at example.com
>
> You can also group your log arguments (again just like shell):
>
> 2012-12-07T12:06:11-05:00 server1 program_name: "these are all one arg"
>
> The routelog(1) program emits to stdout, so you can pipline your log
> processing inside a single command:
>
> routelog mail-errors.rules /var/log/*.log | bzip2 > todays-logs.`date
> +%s`.bz2
>
> It's easily implemented by any log rotation program, via cron, or using a
> queue system like fsq (github.com/axialmarket/fsq ... also available via
> pip) in concert with log rotation (which is how we use it).
>
> Any thoughts, suggestions or improvements (or ports Makefiles ... ;)) are
> much appreciated.
>
> --
> regards,
> matt
>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org/mailman/listinfo/talk
>