Valuecounter aggregator plugin #3523

piotr1212 · 2017-11-29T10:02:21Z

A valuecounter aggregator plugin to count values in specified fields.
Emits the aggregated count of the values.

A usecase for the valuecounter plugin is when you are processing a HTTP access
log (with the logparser input) and want to count the HTTP status codes.

The fields which will be counted must be configured with the field_names
configuration directive. When no field_names is provided the plugin will not
count any fields. The results are emitted in fields in the format:
originalfieldname_fieldvalue = count.

See also: https://community.influxdata.com/t/telegraf-count-field-values-processing-http-response-codes/3214/5

Required for all PRs:

Signed CLA.
Associated README.md updated.
Has appropriate unit tests.

Emits the aggregated count of the values.

slogar · 2018-01-08T08:57:25Z

This looks like a feature that we need. Can you please merge this one so we can deploy this?

danielnelson

Can you make these tweaks?:

danielnelson · 2018-04-12T23:22:53Z

plugins/aggregators/valuecounter/valuecounter.go

+
+type ValueCounter struct {
+	cache       map[uint64]aggregate
+	Field_names []string


Call this Fields, which will match with some other aggregator work.

Strictly speaking, we could probably use the normal filtering options, since this is essentially fieldpass, but I think it is important to require this to be opt in.

danielnelson · 2018-04-12T23:23:13Z

plugins/aggregators/valuecounter/valuecounter.go

+  ## aggregator and will not get sent to the output plugins.
+  drop_original = false
+  ## The fields for which the values will be counted
+  field_names = ""


Should be a list: fields = []

danielnelson · 2018-04-12T23:30:26Z

plugins/aggregators/valuecounter/valuecounter.go

+
+	// Check if this metric has fields which we need to count, if so increment
+	// the count.
+	for fk, fv := range in.Fields() {


I wonder if we should skip floats? One thing that would be bad is if there is a frequently changing float value, which could result in many new fields.

If we don't disallow this, maybe we should add a warning in the README.

good suggestion, will check if I can implement this

danielnelson · 2018-04-12T23:43:36Z

plugins/aggregators/valuecounter/valuecounter.go

+	}
+}
+
+// Emmit the counters


Hi Daniel
My requirement is to filter the logs by response code. If the response code is 500 then only we have to insert in the influx db

Please suggest me the approach

danielnelson · 2018-04-12T23:45:19Z

plugins/aggregators/valuecounter/README.md

+# ValueCounter Aggregator Plugin
+
+The valuecounter plugin counts the occurance of values in fields and emmits the
+counter once every 'preriod' seconds.


typo: period

piotr1212 · 2018-05-30T20:25:44Z

Sorry completely missed the notification about your review. Will check it asap.

piotr1212 · 2018-05-31T13:59:53Z

Apart from the float check the suggestions are implemented.

For some reason when running head I don't see events from inputs.logparser going through the aggregators at all. I do see events from other plugins passing by. I'm not sure if this is a bug or that I am doing something wrong here..

danielnelson · 2018-05-31T18:01:12Z

Most likely the logs you are parsing are old, and only the metrics with a timestamp in the current period are added to the aggregators. There is also a minor bug with service inputs adding metrics too early right at startup, sometimes I run into it when working with very small test sets.

+add some comments

piotr1212 · 2018-06-08T13:14:21Z

The last commit implements filtering on type as suggested. I've also updated the docs.

danielnelson · 2018-06-13T00:52:18Z

plugins/aggregators/valuecounter/valuecounter.go

+						"Must be an int, string or bool. Ignoring.")
+					continue
+				case uint8, uint16, uint32, uint64, int8,
+					int16, int32, int64, string, bool:


Only need to support int64, uint64, bool, and string.

ok, thanks, updated.

piotr1212 · 2018-06-13T09:02:48Z

I think the failing 'release' test is unrelated

danielnelson · 2018-06-13T18:52:10Z

I think it will clear up if you merge or rebase master.

glinton

lgtm

piotr1212 · 2018-06-19T07:58:49Z

thanks

* origin: (39 commits) Document path tag in tail input Update changelog Added path tag to tail input plugin (influxdata#4292) Run windows tests with -short Fix postfix input handling of multi-level queues (influxdata#4333) Update changelog Add support for comma in logparser timestamp format (influxdata#4311) Update vendoring to dep from gdm (influxdata#4314) Update changelog Add new measurement with results of pgrep lookup to procstat input (influxdata#4307) Update changelog Add valuecounter aggregator plugin (influxdata#3523) Update changelog Update docker input documentation for container status Add container status tag to docker input (influxdata#4259) Drop CI support for Go 1.8 (influxdata#4309) Update changelog Fix selection of tags under nested objects in the JSON parser (influxdata#4284) Update changelog Add owner tag on partitions in burrow input (influxdata#4281) ...

dataviruset · 2018-09-03T11:15:03Z

It would be great if there would only be one field and the counted values would be tagged instead of having multiple fields. Makes it easier to do GROUP BY etc.

danielnelson · 2018-09-04T19:13:08Z

@dataviruset You might want to use the regex and/or converter processors instead. Open a new issue with more details about your use case or post on the InfluxData Community site if you want to discuss further.

dataviruset · 2018-09-05T04:11:57Z

@danielnelson Thanks for the reply. I created a post with more details on the community: https://community.influxdata.com/t/tail-and-count-log-file-with-tags/6484

piotr1212 added 2 commits November 29, 2017 10:52

A valuecounter aggregator plugin to count values in specified fields.

da53b95

Emits the aggregated count of the values.

add valuecounter to README.md

ac6aabe

danielnelson added the new plugin label Apr 12, 2018

danielnelson reviewed Apr 12, 2018

View reviewed changes

danielnelson mentioned this pull request Apr 17, 2018

Add plugin to monitor webserver/nginx log files more easily #3991

Closed

piotr1212 added 3 commits May 31, 2018 13:15

Merge remote-tracking branch 'origin/master' into valuecounter

3981abc

valuecounter: implement suggestions

2a149d4

go fmt

ce22ce4

piotr1212 mentioned this pull request Jun 4, 2018

Don't skip metrics during startup in aggregate phase #4230

Merged

3 tasks

piotr1212 added 2 commits June 4, 2018 14:42

Merge remote-tracking branch 'origin/master' into valuecounter

b69e125

valuecounter: only operate on int/str/bool fields

6c77240

+add some comments

danielnelson reviewed Jun 13, 2018

View reviewed changes

danielnelson requested a review from glinton June 13, 2018 00:53

only check for int64, uint64, bool, and string.

ffcabc3

Merge remote-tracking branch 'origin/master' into valuecounter

e584089

glinton approved these changes Jun 18, 2018

View reviewed changes

danielnelson merged commit 3ad1028 into influxdata:master Jun 19, 2018

maxunt pushed a commit that referenced this pull request Jun 26, 2018

Add valuecounter aggregator plugin (#3523)

a059689

rgitzel pushed a commit to rgitzel/telegraf that referenced this pull request Oct 17, 2018

Add valuecounter aggregator plugin (influxdata#3523)

63652ec

otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019

Add valuecounter aggregator plugin (influxdata#3523)

6ec98af

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Valuecounter aggregator plugin #3523

Valuecounter aggregator plugin #3523

piotr1212 commented Nov 29, 2017 •

edited

Loading

slogar commented Jan 8, 2018

danielnelson left a comment

danielnelson Apr 12, 2018

danielnelson Apr 12, 2018

danielnelson Apr 12, 2018

piotr1212 May 31, 2018

danielnelson Apr 12, 2018

ramyam07 Feb 19, 2022

danielnelson Apr 12, 2018

piotr1212 commented May 30, 2018

piotr1212 commented May 31, 2018

danielnelson commented May 31, 2018

piotr1212 commented Jun 8, 2018

danielnelson Jun 13, 2018

piotr1212 Jun 13, 2018

piotr1212 commented Jun 13, 2018

danielnelson commented Jun 13, 2018

glinton left a comment

piotr1212 commented Jun 19, 2018

dataviruset commented Sep 3, 2018

danielnelson commented Sep 4, 2018

dataviruset commented Sep 5, 2018

Valuecounter aggregator plugin #3523

Valuecounter aggregator plugin #3523

Conversation

piotr1212 commented Nov 29, 2017 • edited Loading

Required for all PRs:

slogar commented Jan 8, 2018

danielnelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piotr1212 commented May 30, 2018

piotr1212 commented May 31, 2018

danielnelson commented May 31, 2018

piotr1212 commented Jun 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piotr1212 commented Jun 13, 2018

danielnelson commented Jun 13, 2018

glinton left a comment

Choose a reason for hiding this comment

piotr1212 commented Jun 19, 2018

dataviruset commented Sep 3, 2018

danielnelson commented Sep 4, 2018

dataviruset commented Sep 5, 2018

piotr1212 commented Nov 29, 2017 •

edited

Loading