Prometheus Output

[oldmantaiter]

I was wondering if it is inline with the project goals/style for me to have an output plugin for prometheus that is a local HTTP server that listens on a configurable address/port for prometheus to poll. Basically it would enable a prometheus client. I have a working implementation, but was wondering if it would be of use before writing tests etc.

[sparrc]

@oldmantaiter So this would be an output plugin that starts up a server with an endpoint for prometheus to query from? Then that server needs to accept flush writes from Telegraf while also accepting requests from prometheus?

That definitely sounds like it would be useful, I know @brian-brazil would be interested.

I don't think that it's possible, however, with the current Output interface. We will need to create a ServiceOutput interface similar to the ServicePlugin interface.

[oldmantaiter]

@sparrc Correct, the current implementation does the following:
- Connect: to spawn the HTTP server with the prometheus handler (requires github.com/prometheus/client_golang/prometheus)
- Write: set prometheus gauge metrics for each point that comes through

I will put my current code in my fork, although I am still tweaking it.

[brian-brazil]

I already wrote up the code for this, see master...brian-brazil:prometheus and #15

• Write: set prometheus gauge metrics for each point that comes through

You should produce untyped metrics at scrape time (untyped as we don't know if they're counters or gauges). This keeps the timestamps correct.

[sparrc]

@brian-brazil This is different because we'd be creating a generic interface for outputs to provide a service, rather than having prometheus being integrated directly into the agent.

@oldmantaiter let me know if you need assistance or want me to add some of the generic interface code. My thought was that we could make a generic ServiceOutput interface similar to ServicePlugin: https://github.com/influxdb/telegraf/blob/master/plugins/registry.go#L32-L48

This ServiceOutput would then have it's Start() method called when the agent is starting up, it would be similar to how the ServicePlugin Start() method gets called: https://github.com/influxdb/telegraf/blob/master/agent.go#L437-L444

[oldmantaiter]

@sparrc I'll take a look.

@brian-brazil I had considered supporting counters, but from a really brief glance had not really seen any metrics that would warrant it. If that is not the case, I can look into adding support.

[brian-brazil]

@brian-brazil I had considered supporting counters, but from a really brief glance had not really seen any metrics that would warrant it. If that is not the case, I can look into adding support

It's a question of what the plugin produces, for example /proc/stat is almost all counters. As you don't want to go down the rabbit hole of figuring out which metrics are of which type so that you get the metadata correct (which Prometheus currently doesn't use), just make everything untyped.

[oldmantaiter]

@brian-brazil You are absolutely correct, I was thinking of metrics in terms of how our current in-house implementation (which we are replacing with telegraf) presents them (counters are converted to gauges). I will make adjustments.

EDIT: Replied before my morning coffee, I will move the current GaugeVec to UntypedVec

[oldmantaiter]

@sparrc I have not changed it to be a new ServiceOutput interface yet, that will be in my next commit.

@brian-brazil The code I had written required an update to the prometheus/client_golang library that was in godep, so I had to change the current prometheus plugin to use the new functions. I have tested locally and it looks to be working, but I would like your eyes thoughts on the change - oldmantaiter@4b06bb3#diff-55c12536a8b6055b2d437db596976dbfL5

[brian-brazil]

@oldmantaiter That all looks okay from a quick peek, I left some small comments.

[oldmantaiter]

@brian-brazil Great, thanks, updated to use port 9126 and removed some useless comments.

I'll keep plugging away on the interface for ServiceOutput. At this point, would it make more sense for me to track this in a PR or continue updating this issue?

[sparrc]

Thanks a bunch @oldmantaiter, send me an email cameron (at) influxdb.com if you'd like to do a google hangout

Feel free to open a PR anytime you'd like us to review some code, doesn't have to be finished