Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloudwatch input plugin doesn't work with prometheus output metric_version 2 #6926

Closed
surprisingb opened this issue Jan 20, 2020 · 11 comments · Fixed by #7056
Closed

Cloudwatch input plugin doesn't work with prometheus output metric_version 2 #6926

surprisingb opened this issue Jan 20, 2020 · 11 comments · Fixed by #7056
Assignees
@danielnelson
Labels
area/prometheus bug unexpected problem or unintended behavior
Milestone
1.13.4

Comments

@surprisingb
Copy link

Since the introduction of the metric_version = 2 configuration the cloudwatch input plugin doesn't produce any metric.

Relevant telegraf.conf:

#
# OUTPUTS:
#
[[outputs.prometheus_client]]
listen = ":9273"
#
# INPUTS:
#
[[inputs.cloudwatch]]
delay = "5m"
interval = "1m"
namespace = "AWS/ApplicationELB"
period = "1m"
region = "eu-central-1"
[[inputs.cloudwatch.metrics]]
names = ["HealthyHostCount", "UnHealthyHostCount"]
statistic_include = ["average"]
[[inputs.cloudwatch.metrics.dimensions]]
name = "LoadBalancer"
value = "*"
[[inputs.cloudwatch.metrics.dimensions]]
name = "TargetGroup"
value = "*"
[[inputs.cloudwatch.metrics]]
names = ["TargetResponseTime"]
statistic_include = ["average"]
[[inputs.cloudwatch.metrics.dimensions]]
name = "LoadBalancer"
value = "*"
[[inputs.cloudwatch.metrics]]
names = ["RequestCount", "HTTPCode_Target_2XX_Count", "HTTPCode_Target_4XX_Count", "HTTPCode_Target_5XX_Count", "HTTPCode_ELB_4XX_Count", "HTTPCode_ELB_5XX_Count"]
statistic_include = ["sum"]
[[inputs.cloudwatch.metrics.dimensions]]
name = "LoadBalancer"
value = "*"
[[inputs.cloudwatch.metrics]]
names = ["RequestCountPerTarget"]
statistic_include = ["sum"]
[[inputs.cloudwatch.metrics.dimensions]]
name = "TargetGroup"
value = "*"
[[inputs.cloudwatch]]
delay = "5m"
interval = "1m"
namespace = "AWS/AutoScaling"
period = "1m"
region = "eu-central-1"
[[inputs.cloudwatch.metrics]]
names = ["GroupDesiredCapacity", "GroupInServiceInstances", "GroupTerminatingInstances", "GroupPendingInstances", "GroupStandbyInstances", "GroupTotalInstances", "GroupMaxSize"]
statistic_include = ["average"]
[[inputs.cloudwatch.metrics.dimensions]]
name = "AutoScalingGroupName"
value = "*"

System info:

# telegraf --version
Telegraf 1.13.1 (git: HEAD 0c175724)

# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Steps to reproduce:

  1. Add the configuration metric_version = 2 in the prometheus output section as recommended since the latest telegraf version

Expected behavior:

The telegraf prometheus output should be the same as with the metric_version = 1 configuration.

Actual behavior:

The telegraf output is entirely missing the cloudwatch input plugin metrics.

Additional info:

No message of any sort is logged by telegraf.
@danielnelson
Copy link
Contributor

Can you add the output of this command?

telegraf --input-filter cloudwatch --test

@danielnelson danielnelson added area/prometheus bug unexpected problem or unintended behavior labels Jan 21, 2020
@surprisingb
Copy link
Author

Telegraf has been updated to version 1.13.2 last night, but I still have the same issue.
Here the output of the tests:

# grep -A2 prometheus_client telegraf_v1.conf            
[[outputs.prometheus_client]]
listen = ":9273"
metric_version = 1
# telegraf --input-filter cloudwatch --test --config telegraf_v1.conf 
2020-01-23T07:34:54Z I! Starting Telegraf 1.13.2
2020-01-23T07:34:54Z W! [outputs.prometheus_client] Use of deprecated configuration: metric_version = 1; please update to metric_version = 2
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb1,target_group=targetgroup/target-group1 healthy_host_count_average=2,un_healthy_host_count_average=0 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb2,target_group=targetgroup/target-group2 healthy_host_count_average=2,un_healthy_host_count_average=0 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb3,target_group=targetgroup/target-group3 healthy_host_count_average=2,un_healthy_host_count_average=0 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb4,target_group=targetgroup/target-group4 healthy_host_count_average=3,un_healthy_host_count_average=0 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb2 http_code_target_2xx_count_sum=1368,http_code_target_4xx_count_sum=4,request_count_sum=1372,target_response_time_average=0.05737875364431486 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb1 http_code_target_2xx_count_sum=761,request_count_sum=761,target_response_time_average=0.01827387779237845 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb4 http_code_elb_4xx_count_sum=4,http_code_target_2xx_count_sum=243,request_count_sum=247,target_response_time_average=0.01861504526748971 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb3 http_code_target_2xx_count_sum=4069,http_code_target_4xx_count_sum=8,request_count_sum=4077,target_response_time_average=0.014352314692175619 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group4 request_count_per_target_sum=82.33333400000001 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group3 request_count_per_target_sum=2038.5 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group2 request_count_per_target_sum=686 1579764480000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group1 request_count_per_target_sum=380.5 1579764480000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg1,host=prometheus_host group_desired_capacity_average=3,group_in_service_instances_average=3,group_max_size_average=10,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=3 1579764480000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg2,host=prometheus_host group_desired_capacity_average=2,group_in_service_instances_average=2,group_max_size_average=10,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=2 1579764480000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg3,host=prometheus_host group_desired_capacity_average=2,group_in_service_instances_average=2,group_max_size_average=10,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=2 1579764480000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg4n,host=prometheus_host group_desired_capacity_average=2,group_in_service_instances_average=2,group_max_size_average=6,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=2 1579764480000000000

# grep -A2 prometheus_client telegraf_v2.conf  
[[outputs.prometheus_client]]
listen = ":9273"
metric_version = 2
# telegraf --input-filter cloudwatch --test --config telegraf_v2.conf  
2020-01-23T07:35:12Z I! Starting Telegraf 1.13.2
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb1,target_group=targetgroup/target-group1 healthy_host_count_average=2,un_healthy_host_count_average=0 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb2,target_group=targetgroup/target-group2 healthy_host_count_average=2,un_healthy_host_count_average=0 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb3,target_group=targetgroup/target-group3 healthy_host_count_average=2,un_healthy_host_count_average=0 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb4,target_group=targetgroup/target-group4 healthy_host_count_average=3,un_healthy_host_count_average=0 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb2 http_code_target_2xx_count_sum=1319,http_code_target_4xx_count_sum=2,request_count_sum=1321,target_response_time_average=0.055556818319454956 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb1 http_code_target_2xx_count_sum=811,request_count_sum=811,target_response_time_average=0.01854795067817509 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb4 http_code_elb_4xx_count_sum=8,http_code_target_2xx_count_sum=248,request_count_sum=256,target_response_time_average=0.021855149193548386 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,load_balancer=app/alb3 http_code_target_2xx_count_sum=3589,http_code_target_4xx_count_sum=4,request_count_sum=3593,target_response_time_average=0.014130970219871973 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group4 request_count_per_target_sum=85.33333499999999 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group3 request_count_per_target_sum=1796.5 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group2 request_count_per_target_sum=660.5 1579764540000000000
> cloudwatch_aws_application_elb,host=prometheus_host,target_group=targetgroup/target-group1 request_count_per_target_sum=405.5 1579764540000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg1,host=prometheus_host group_desired_capacity_average=3,group_in_service_instances_average=3,group_max_size_average=10,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=3 1579764540000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg2,host=prometheus_host group_desired_capacity_average=2,group_in_service_instances_average=2,group_max_size_average=10,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=2 1579764540000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg3,host=prometheus_host group_desired_capacity_average=2,group_in_service_instances_average=2,group_max_size_average=10,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=2 1579764540000000000
> cloudwatch_aws_auto_scaling,auto_scaling_group_name=asg4n,host=prometheus_host group_desired_capacity_average=2,group_in_service_instances_average=2,group_max_size_average=6,group_pending_instances_average=0,group_standby_instances_average=0,group_terminating_instances_average=0,group_total_instances_average=2 1579764540000000000

The outputs seem similar to me, but I'm still not getting any cloudwatch metric in the metric HTTP page nor in prometheus when metric_version is set to 2.

@danielnelson
Copy link
Contributor

I not able to replicate, here is what I'm doing:

  • Copy the line protocol into a file tmp.lp (without the leading > )
  • Start telegraf with this configuration: 
    [[inputs.file]]
  files = ["tmp.lp"]
  data_format = "influx"

[[outputs.prometheus_client]]
  listen = ":9273"
  metric_version = 2
  collectors_exclude = ["gocollector", "process"]

Check the output, this is only showing the first metric but the others are there too though.

$ curl http://localhost:9273/metrics
# HELP cloudwatch_aws_application_elb_healthy_host_count_average Telegraf collected metric
# TYPE cloudwatch_aws_application_elb_healthy_host_count_average untyped
cloudwatch_aws_application_elb_healthy_host_count_average{host="prometheus_host",load_balancer="app/alb1",target_group="targetgroup/target-group1"} 2
cloudwatch_aws_application_elb_healthy_host_count_average{host="prometheus_host",load_balancer="app/alb2",target_group="targetgroup/target-group2"} 2
cloudwatch_aws_application_elb_healthy_host_count_average{host="prometheus_host",load_balancer="app/alb3",target_group="targetgroup/target-group3"} 2
cloudwatch_aws_application_elb_healthy_host_count_average{host="prometheus_host",load_balancer="app/alb4",target_group="targetgroup/target-group4"} 3

@surprisingb
Copy link
Author

surprisingb commented Jan 24, 2020 •
edited
Loading

I tried to do the same, no output at all:

# cat telegraf_v2.conf 
[[inputs.file]]
  files = ["tmp.lp"]
  data_format = "influx"

[[outputs.prometheus_client]]
  listen = ":9274"
  metric_version = 2
  collectors_exclude = ["gocollector", "process"]
# telegraf --config telegraf_v2.conf 
2020-01-24T13:03:27Z I! Starting Telegraf 1.13.2
2020-01-24T13:03:27Z I! Loaded inputs: file
2020-01-24T13:03:27Z I! Loaded aggregators: 
2020-01-24T13:03:27Z I! Loaded processors: 
2020-01-24T13:03:27Z I! Loaded outputs: prometheus_client
2020-01-24T13:03:27Z I! Tags enabled: host=prometheus_host
2020-01-24T13:03:27Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"prometheus_host", Flush Interval:10s
2020-01-24T13:03:27Z I! [outputs.prometheus_client] Listening on http://[::]:9274/metrics

$ curl localhost:9274/metrics
$

Telegraf is installed via the influxdata Yum repository, still at:

$ telegraf --version
Telegraf 1.13.2 (git: HEAD 6dad859d)

Thanks for your help.

@surprisingb
Copy link
Author

I just tried with 2 different container, a Centos 7 and a Debian 9, same telegraf version, same input file: no output again.

@danielnelson
Copy link
Contributor

Can you enable the internal input and show the response from the prometheus_client output?

@surprisingb
Copy link
Author

There's the output of the internal input plugin with metric_version = 2:

# HELP internal_agent_gather_errors Telegraf collected metric
# TYPE internal_agent_gather_errors untyped
internal_agent_gather_errors{go_version="1.13.5",host="prometheus_host",version="1.13.2"} 0
# HELP internal_agent_metrics_dropped Telegraf collected metric
# TYPE internal_agent_metrics_dropped untyped
internal_agent_metrics_dropped{go_version="1.13.5",host="prometheus_host",version="1.13.2"} 0
# HELP internal_agent_metrics_gathered Telegraf collected metric
# TYPE internal_agent_metrics_gathered untyped
internal_agent_metrics_gathered{go_version="1.13.5",host="prometheus_host",version="1.13.2"} 7173
# HELP internal_agent_metrics_written Telegraf collected metric
# TYPE internal_agent_metrics_written untyped
internal_agent_metrics_written{go_version="1.13.5",host="prometheus_host",version="1.13.2"} 6405
# HELP internal_gather_errors Telegraf collected metric
# TYPE internal_gather_errors untyped
internal_gather_errors{host="prometheus_host",input="cloudwatch",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="cpu",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="disk",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="diskio",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="internal",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="kernel",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="mem",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="net",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="processes",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="procstat",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="prometheus",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="puppetagent",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="swap",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="system",version="1.13.2"} 0
internal_gather_errors{host="prometheus_host",input="unbound",version="1.13.2"} 0
# HELP internal_gather_gather_time_ns Telegraf collected metric
# TYPE internal_gather_gather_time_ns untyped
internal_gather_gather_time_ns{host="prometheus_host",input="cloudwatch",version="1.13.2"} 2.45089912e+08
internal_gather_gather_time_ns{host="prometheus_host",input="cpu",version="1.13.2"} 122602
internal_gather_gather_time_ns{host="prometheus_host",input="disk",version="1.13.2"} 441057
internal_gather_gather_time_ns{host="prometheus_host",input="diskio",version="1.13.2"} 148750
internal_gather_gather_time_ns{host="prometheus_host",input="internal",version="1.13.2"} 246036
internal_gather_gather_time_ns{host="prometheus_host",input="kernel",version="1.13.2"} 95341
internal_gather_gather_time_ns{host="prometheus_host",input="mem",version="1.13.2"} 142501
internal_gather_gather_time_ns{host="prometheus_host",input="net",version="1.13.2"} 356006
internal_gather_gather_time_ns{host="prometheus_host",input="processes",version="1.13.2"} 1.2862728e+07
internal_gather_gather_time_ns{host="prometheus_host",input="procstat",version="1.13.2"} 1.3657953e+07
internal_gather_gather_time_ns{host="prometheus_host",input="prometheus",version="1.13.2"} 2.9378475e+07
internal_gather_gather_time_ns{host="prometheus_host",input="puppetagent",version="1.13.2"} 377760
internal_gather_gather_time_ns{host="prometheus_host",input="swap",version="1.13.2"} 200306
internal_gather_gather_time_ns{host="prometheus_host",input="system",version="1.13.2"} 1.180735e+06
internal_gather_gather_time_ns{host="prometheus_host",input="unbound",version="1.13.2"} 5.8307531e+07
# HELP internal_gather_metrics_gathered Telegraf collected metric
# TYPE internal_gather_metrics_gathered untyped
internal_gather_metrics_gathered{host="prometheus_host",input="cloudwatch",version="1.13.2"} 31
internal_gather_metrics_gathered{host="prometheus_host",input="cpu",version="1.13.2"} 9
internal_gather_metrics_gathered{host="prometheus_host",input="disk",version="1.13.2"} 4
internal_gather_metrics_gathered{host="prometheus_host",input="diskio",version="1.13.2"} 8
internal_gather_metrics_gathered{host="prometheus_host",input="internal",version="1.13.2"} 77
internal_gather_metrics_gathered{host="prometheus_host",input="kernel",version="1.13.2"} 4
internal_gather_metrics_gathered{host="prometheus_host",input="mem",version="1.13.2"} 4
internal_gather_metrics_gathered{host="prometheus_host",input="net",version="1.13.2"} 8
internal_gather_metrics_gathered{host="prometheus_host",input="processes",version="1.13.2"} 4
internal_gather_metrics_gathered{host="prometheus_host",input="procstat",version="1.13.2"} 20
internal_gather_metrics_gathered{host="prometheus_host",input="prometheus",version="1.13.2"} 6972
internal_gather_metrics_gathered{host="prometheus_host",input="puppetagent",version="1.13.2"} 4
internal_gather_metrics_gathered{host="prometheus_host",input="swap",version="1.13.2"} 8
internal_gather_metrics_gathered{host="prometheus_host",input="system",version="1.13.2"} 12
internal_gather_metrics_gathered{host="prometheus_host",input="unbound",version="1.13.2"} 8
# HELP internal_memstats_alloc_bytes Telegraf collected metric
# TYPE internal_memstats_alloc_bytes untyped
internal_memstats_alloc_bytes{host="prometheus_host"} 1.0576648e+07
# HELP internal_memstats_frees Telegraf collected metric
# TYPE internal_memstats_frees untyped
internal_memstats_frees{host="prometheus_host"} 1.036135e+06
# HELP internal_memstats_heap_alloc_bytes Telegraf collected metric
# TYPE internal_memstats_heap_alloc_bytes untyped
internal_memstats_heap_alloc_bytes{host="prometheus_host"} 1.0576648e+07
# HELP internal_memstats_heap_idle_bytes Telegraf collected metric
# TYPE internal_memstats_heap_idle_bytes untyped
internal_memstats_heap_idle_bytes{host="prometheus_host"} 5.3075968e+07
# HELP internal_memstats_heap_in_use_bytes Telegraf collected metric
# TYPE internal_memstats_heap_in_use_bytes untyped
internal_memstats_heap_in_use_bytes{host="prometheus_host"} 1.3312e+07
# HELP internal_memstats_heap_objects Telegraf collected metric
# TYPE internal_memstats_heap_objects untyped
internal_memstats_heap_objects{host="prometheus_host"} 126779
# HELP internal_memstats_heap_released_bytes Telegraf collected metric
# TYPE internal_memstats_heap_released_bytes untyped
internal_memstats_heap_released_bytes{host="prometheus_host"} 4.919296e+07
# HELP internal_memstats_heap_sys_bytes Telegraf collected metric
# TYPE internal_memstats_heap_sys_bytes untyped
internal_memstats_heap_sys_bytes{host="prometheus_host"} 6.6387968e+07
# HELP internal_memstats_mallocs Telegraf collected metric
# TYPE internal_memstats_mallocs untyped
internal_memstats_mallocs{host="prometheus_host"} 1.162914e+06
# HELP internal_memstats_num_gc Telegraf collected metric
# TYPE internal_memstats_num_gc untyped
internal_memstats_num_gc{host="prometheus_host"} 11
# HELP internal_memstats_pointer_lookups Telegraf collected metric
# TYPE internal_memstats_pointer_lookups untyped
internal_memstats_pointer_lookups{host="prometheus_host"} 0
# HELP internal_memstats_sys_bytes Telegraf collected metric
# TYPE internal_memstats_sys_bytes untyped
internal_memstats_sys_bytes{host="prometheus_host"} 7.1762168e+07
# HELP internal_memstats_total_alloc_bytes Telegraf collected metric
# TYPE internal_memstats_total_alloc_bytes untyped
internal_memstats_total_alloc_bytes{host="prometheus_host"} 6.550252e+07
# HELP internal_write_buffer_limit Telegraf collected metric
# TYPE internal_write_buffer_limit untyped
internal_write_buffer_limit{alias="",host="prometheus_host",output="prometheus_client",version="1.13.2"} 10000
# HELP internal_write_buffer_size Telegraf collected metric
# TYPE internal_write_buffer_size untyped
internal_write_buffer_size{alias="",host="prometheus_host",output="prometheus_client",version="1.13.2"} 767
# HELP internal_write_errors Telegraf collected metric
# TYPE internal_write_errors untyped
internal_write_errors{host="prometheus_host",output="prometheus_client",version="1.13.2"} 0
# HELP internal_write_metrics_added Telegraf collected metric
# TYPE internal_write_metrics_added untyped
internal_write_metrics_added{alias="",host="prometheus_host",output="prometheus_client",version="1.13.2"} 7172
# HELP internal_write_metrics_dropped Telegraf collected metric
# TYPE internal_write_metrics_dropped untyped
internal_write_metrics_dropped{alias="",host="prometheus_host",output="prometheus_client",version="1.13.2"} 0
# HELP internal_write_metrics_filtered Telegraf collected metric
# TYPE internal_write_metrics_filtered untyped
internal_write_metrics_filtered{host="prometheus_host",output="prometheus_client",version="1.13.2"} 0
# HELP internal_write_metrics_written Telegraf collected metric
# TYPE internal_write_metrics_written untyped
internal_write_metrics_written{alias="",host="prometheus_host",output="prometheus_client",version="1.13.2"} 6405
# HELP internal_write_write_time_ns Telegraf collected metric
# TYPE internal_write_write_time_ns untyped
internal_write_write_time_ns{host="prometheus_host",output="prometheus_client",version="1.13.2"} 3.561601e+06

@lbatalha
Copy link

lbatalha commented Feb 20, 2020 •
edited
Loading

Hitting the same issue, using Telegraf 1.13.3 on Ubuntu 18.04 from Influx repos.
When executing telegraf --input-filter cloudwatch --test with the config I see the cloudwatch metrics in line format.

InfluxDB output plugin correctly submits the metrics to InfluxDB, but prometheus_client output plugin does not display any at the metrics endpoint.

@danielnelson
Copy link
Contributor

I think I figured out the cause of this, in metric_version = 1 we are using the "write" time when calculating when to expire metrics. In metric_version = 2 we are using the actual metric time instead. For metrics like Cloudwatch that are delayed somewhat, this difference can cause them to be immediately expired with the default of 60s.

Try setting expiration_interval = 60m or higher, does this make them appear in the output?

@danielnelson danielnelson self-assigned this Feb 20, 2020
@danielnelson danielnelson added this to the 1.13.4 milestone Feb 20, 2020
@danielnelson danielnelson mentioned this issue Feb 20, 2020
Merged
3 tasks
@lbatalha
Copy link

Indeed that seems to work! Thank you! This should be a good holdover until 1.13.4

@surprisingb
Copy link
Author

Can confirm it works, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danielnelson danielnelson

Labels
area/prometheus bug unexpected problem or unintended behavior
Projects
None yet
Milestone
1.13.4
Development

Successfully merging a pull request may close this issue.

Use add time for prometheus expiration calculation influxdata/telegraf
3 participants
@danielnelson @lbatalha @surprisingb