[nycbug-talk] Statistical Monitoring

[Jesse Callaway]

On Wed, Nov 5, 2008 at 11:03 AM, Matt Juszczak <matt at atopia.net> wrote:
>>>> Once you get ganglia up and running, all it will do is provide you
>>>> with statistics and it will be up to you to figure out a way to
>>>> collect and store them and then do something meaningful with them.
>
> The problem I have is I want TONS of graphs.  I want to graph our load
> balancer, our firewall, our CPU usage for specific processes across
> servers (apache, memcache, mysql, etc.), memory usage (free/available),
> mysql statistics (threads running, queries running, long running queries,
> average query time, seconds behind master, etc.), and much much more.  If
> I have all of these statistics being reported (and graphed), then is this
> something that reliably, a pull method can perform well?  I've used SNMP a
> lot to gather basic statistics, but I doubt I'd be able to get SNMP to
> broadcast what the current queries per second are on the local MySQL
> server easily (I know its possible - there's an SNMP module for MySQL, but
> I doubt its trivial).  Wouldn't something like this be better as a script
> running on ALL servers to gather the statistics and push those statistics
> to a centralized daemon of sorts running on the server?
>
> But since I also need to graph things that are snmp-based (for instance,
> our load balancer information can only be obtained via snmp), my thoughts
> are that using cacti is most likely the best option, but I'd have to use
> the custom-graph-with-scripts option more often.  Or, like I asked,
> perhaps using ganglia to push the statistics, and then running a script on
> the cacti server to convert the ganglia data into graphs?
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org/mailman/listinfo/talk
>

Most systems let you mix up SNMP and whatever the heck else you can
get some numbers out of. I don't think Cacti is limited to data from
SNMP at all. And I know that Nagios is not.
With Nagios you can push you can pull you can wait for data to come
in. You can freak if the data does not come in... there's lots of
options. Although it was mostly built to send you alerts if something
doesn't look right to it, you can extend it to give you nice trend
graphs. I graph a bunch of MySQL information with it
(Uptime_since_flush_status, Last_query_cost, Connections,
Slow_queries, Open_tables, Questions, Threads_running,
Innodb_row_lock_time_avg) using the pnp4nagios plugin.

Looking back I should have done Nagios and Cacti separately. Just
monitor a few things with Nagios to determine if a critical service is
UP or DOWN. If you want to get real simple, check out sysmon. George
loves it, but for some reason he's not piping up about it. It's good
for this.
For all the pretty graphs use Cacti and a plugin to layout the graphs
in a nice way.

-jesse