jeff blaine - Latest Comments

Re: Any-Metric Graphing with Graphite and Syslog

Jeff Blaine — Fri, 27 Apr 2012 16:06:58 -0000

Hey Mark, by all means feel free to work with MetricLogster. I am not attached to my particular syntax at all and will gladly adapt to any changes you make there.

Re: Any-Metric Graphing with Graphite and Syslog

Mark Crossfield — Tue, 24 Apr 2012 06:00:05 -0000

We've been using logster and Graphite for around nine months now and over the weekend I had a similar idea to you, which I've implemented independently. I'm kicking myself for not thinking of it sooner as I've been frustrated by having to show a number of people how to write Python parsers :)

Anyway, I'm keen to contribute what I've done back to the project, but I wanted to bounce it off you first. I've done something similar to allow arbitrary counter metrics to be incremented, but also (in the same vein as StatsD) to allow times/durations to be recorded. The difference with the durations is that the median and 90th percentile are calculated and pushed into Graphite rather than the total. I've written a stats helper to calculate these as I didn't want to introduce a dependency on an external Python package (and it's not my first language).

My intended producer of these logs is Log4J like logs, so the syntax of my log statements has differed. I'm not sure how attached to your syntax you are. What I'm coming to is: should I contribute to your parser or create a whole new one?

Re: Any-Metric Graphing with Graphite and Syslog

ndietsch — Fri, 20 Apr 2012 23:00:30 -0000

That is a really interesting implementation and could prove very useful for people who already have syslog punching through firewalls.

Re: Any-Metric Graphing with Graphite and Syslog

Jeff Blaine — Tue, 10 Apr 2012 10:05:23 -0000

Oof! I mistakenly attributed Graphite to Etsy in the original version of this post.

Orbitz developed Graphite and released it, not Etsy.

I have updated the post.

Re: Any-Metric Graphing with Graphite and Syslog

Jeff Blaine — Sat, 07 Apr 2012 14:27:10 -0000

Thanks for the bzr info.

Re: Any-Metric Graphing with Graphite and Syslog

Jeff Blaine — Sat, 07 Apr 2012 14:25:11 -0000

Scott, I said "I like using, when possible, what is widely available instead of distributing new package or library dependencies to all nodes in our infrastructure."

Where did I install a new script on all of our nodes?

Re: Any-Metric Graphing with Graphite and Syslog

Ranjib Dey — Sat, 07 Apr 2012 08:47:00 -0000

great post. I'll be employing this strategy to gather request/response time from apache proxy logs.
thanks a ton

Re: Any-Metric Graphing with Graphite and Syslog

scott — Mon, 02 Apr 2012 18:25:22 -0000

hey man, cool post. two things:

1. You said you didn't want to have to install another package, but ...you installed another script? Don't see much difference.
2. You can submit "pull requests" to graphite without joining graphite-dev. In bazaar it works a bit differently - you just branch the repo and open a ticket requesting a merge.

Re: Any-Metric Graphing with Graphite and Syslog

Clif Smith — Sat, 31 Mar 2012 17:30:58 -0000

Great job!

Re: Specialized Syslog Collector for Metrics via Syslog

Eric Lindvall — Mon, 26 Mar 2012 20:21:47 -0000

I haven't had a chance to blog about it yet, but here's the missing link:

https://github.com/eric/metrik...

It's a webhook for Papertrail (https://papertrailapp.com/ ) that I wrote to take the output of the Metriks Logger reporter (Metriks::Reporter::Logger, documented here: https://github.com/eric/metrik... ) and sends it to Librato Metrics (http://metrics.librato.com/ ).

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 18:19:32 -0000

I hadn't considered this, Andrew. Great idea. If I wanted, as you said, I could keep the parser-->Graphite part completely out of the rsyslog code.

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 18:16:52 -0000

Thanks for the link! I'll be watching this later tonight.

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 18:15:39 -0000

Thanks for the reply, Nathan. Anything we implement will certainly be in stages, increasing in metric count and frequency over time, while taking note of the impact.

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 18:11:41 -0000

Thanks Jason. Nice to hear it's being done and is well-liked. I hadn't considered Logstash as we use Splunk and like it, but it may very well be that we'll use it just as the Graphite writer role.

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 18:08:46 -0000

Thanks for the reply, Brian. I'll definitely dig into Logster.

I have in fact seen Joe's statsd server implementation index. I commented there with one implementation that he put on the list ;) However, it is a good reminder that I need to make sure I get myself up to speed with all of the various offerings in this problem domain. Maybe it's time to revisit the "monitoringsucks" github repo as well.

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 18:04:26 -0000

Thanks for the reply, Spike.

I will definitely look into Logster and consider some ways to implement what I am thinking about with it. We have no special feelings toward rsyslog, a plugin for it (or whatever terminology they may use) was just a first idea for how something like this might be done. I didn't know about Logster.

You'll note that I played around with this idea in the past: http://www.kickflop.net/blog/2...

Sadly (?), I didn't really take it any further due to Splunk not providing zero-configuration graphs + dashboards like Graphite-web does. There is also the question of whether or not it really makes sense *for us* to be loading down the host with Splunk indexing of a large volume of metric data. Additionally with Splunk, the product cost is based on the amount of daily indexed data, so you end up paying for metrics. [ Don't get me wrong, we really like Splunk and have been using it for 4 years. ]

Anyway, I'm rambling now. Thanks again!

Re: Specialized Syslog Collector for Metrics via Syslog

Jeff Blaine — Sun, 25 Mar 2012 17:39:21 -0000

Troy, thanks for the reply. Metriks seems to me to be an improved type of StatsD but with multiple possible output destinations instead of just Graphite. I don't see where syslog plays any real role in Metriks as the transport mechanism.
Am I overlooking something?

Re: Specialized Syslog Collector for Metrics via Syslog

Troy — Sun, 25 Mar 2012 15:43:28 -0000

My cohort Eric does something similar. He's released it as a Ruby gem called metriks: http://bitmonkey.net/post/1885... (gem https://github.com/eric/metrik....

He's using it to send metrics about Papertrail's operations to Librato Metrics and graphite. Aside from sending metrics elsewhere, the Ruby process has access to the computed rate and it can update the process name. Having ps show "someworker: 123/second" has been handy.

Re: Specialized Syslog Collector for Metrics via Syslog

Spike Morelli — Sun, 25 Mar 2012 05:00:02 -0000

Hey Jeff,

definitely a sound idea and something I've look at in the past. I've actually done something similar with syslong-ng piping to a script, but it won't scale. The alternative is to rewrite something that understands syslog messages, but then you're back to running non standard components in your infra even tho you're limiting the surface to aggregating nodes.

About a logster-alike solution, which basically means tailing log files, we've had that scaling pretty well using a daemon mode and is a pretty solid choice. Depending how hard you wanna push the envelope a better choice is, like you suggested, to tweak rsyslog which to me would translate to writing a new module. There's already a variety of plugins, to write to DBs, to write to mongo, etc, so I can easily see how you could write something that writes to graphite with statsd like features (rsyslog supports stuff like batching messages which would suit this use case).

So the question to me becomes: why would I want something tightly integrated with rsyslog Vs something generic (logster) that can feed off any text(log) file?

@spikelab

Re: Specialized Syslog Collector for Metrics via Syslog

Brian Henerey — Sun, 25 Mar 2012 03:56:06 -0000

Hi Jeff,
If you want to get metrics out of logs, I second the mentioning of Logster (run on all of your servers) or Logstash (groking against you centralised logs.

Have you seen this? http://joemiller.me/2011/09/21...

Still, syslog-ng approach sounds good and certainly better than installing Node all over the place for no other purpose.

Regards,
Brian

Re: Specialized Syslog Collector for Metrics via Syslog

Jason Dixon — Sun, 25 Mar 2012 00:35:39 -0000

We already do this at Heroku and it's quite popular with our engineers. We're in the process of streamlining many of the emitters and aggregators in our event stream (e.g. syslog is an emitter, Splunk and Graphite are consumers).

In fact you can already do this today via Logster (https://github.com/etsy/logste... and Logstash (http://logstash.net/docs/1.1.0....

Re: Specialized Syslog Collector for Metrics via Syslog

ndietsch — Sat, 24 Mar 2012 16:19:31 -0000

Hi Jeff,

As I said on twitter, your idea is very sound, especially for those of us who already have large syslog infrastructures in play. One thing to consider is that depending on the number of metrics, you may be placing a large amount of load on infrastructure that was not necessarily designed for that purpose.

This may be a particular case, but I saw network routers have a particularly hard time with routing large numbers of UDP packets because they were at the bottom of the filter list, but this was a case of a packet storm, not a well defined infrastructure. I would be monitoring the use of said infrastructure as you scale up the number of metrics you are collecting.

You would want to ensure that the packets remain less than 1536 bytes as I do not think fragmentation would be a particularly pretty scenario either.

Very eager to see how it turns out as it is a very viable alternative to polling SNMP through a firewall.

Nathan Dietsch

Re: Specialized Syslog Collector for Metrics via Syslog

Andrew Fort — Sat, 24 Mar 2012 16:06:49 -0000

You could write to a pipe from syslog and have your parser read that.

Re: Specialized Syslog Collector for Metrics via Syslog

Eric-Olivier Lamey — Sat, 24 Mar 2012 15:18:40 -0000

Hi,
This is basically what is described in this presentation from RubyConf Argentina: https://vimeo.com/38628915

Re: Django: Taking note of ManyToManyField changes

Jeff Blaine — Tue, 24 Jan 2012 16:15:53 -0000

Here's what I ended up doing today to address this as best as I can, even though it is inefficient to do. Note that it is pretty implementation-specific, as we're doing something special in our own post_save callback function (pushing the owner object and its relation data into LDAP as an LDIF replace operation):

def resave_bc_m2m_changed(sender, **kwargs):
    a = kwargs['action']
    if a == 'post_add' or a == 'post_remove':
        kwargs['instance'].save()
        # Don't do anything in pre/post_clear
        # for this effort

m2m_changed.connect(resave_bc_m2m_changed,
           sender=Netgroup.interfaces.through,
           weak=False)
m2m_changed.connect(resave_bc_m2m_changed,
           sender=NetgroupContainer.members.through,
           weak=False)