Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consul input with prometheus output on new versions #5880

Closed
osd42 opened this issue May 20, 2019 · 4 comments
Closed

Consul input with prometheus output on new versions #5880

osd42 opened this issue May 20, 2019 · 4 comments
Labels
area/prometheus bug unexpected problem or unintended behavior

Comments

@osd42
Copy link

osd42 commented May 20, 2019 •
edited
Loading

Relevant telegraf.conf:

# Gather health check statuses from services registered in Consul
[[inputs.consul]]
  ## Consul server address
  address = "localhost:8501"

  ## URI scheme for the Consul server, one of "http", "https"
  scheme = "https"

  ## ACL token used in every request
  # token = ""

  ## HTTP Basic Authentication username and password.
  # username = ""
  # password = ""

  ## Data center to query the health checks from
  # datacenter = ""

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  insecure_skip_verify = true

  ## Consul checks' tag splitting
  # When tags are formatted like "key:value" with ":" as a delimiter then
  # they will be splitted and reported as proper key:value in Telegraf
  # tag_delimiter = ":"

  ## added a custom section to remove error " "
  # bug on 1.10.1 : https://github.com/influxdata/telegraf/issues/5522
  tagexclude = ["[0-9]*", "noc", "", "^_" ]

System info:

centos 7.6 (linux 64)
telegraf 1.9.0 OK
telegraf 1.10.4 KO

Steps to reproduce:

log output :

mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Nomad Client HTTP Check cmpp6 http     95e9bdf85c3c00e9584b4d6104edd6340ce285e8 cmpp4 nomad-client _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Serf Health Status cmpp6      serfHealth cmpp4   passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Nomad Client HTTP Check cmpp7 http     b4e60280df111aa775ff30a1981cc9d9b7ff7e70 cmpp4 nomad-client _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Serf Health Status cmpp7      serfHealth cmpp4   passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Nomad Server RPC Check cmpp4    rpc  8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 cmpp4 nomad _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Nomad Client HTTP Check cmpp8 http     688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa cmpp4 nomad-client _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Nomad Client HTTP Check cmpp9 http     d1cd1cbe5327e848371c937f867644ffbef7456a cmpp4 nomad-client _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ Service 'elasticsearch' check cmpp7      service:elasticsearch cmpp4 elasticsearch elasticsearch passing],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [ secure  Service 'rabbitmqtls' check cmpp4 rabbitmqtls rabbitmqtls    passing service:rabbitmqtls cmpp5],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Client HTTP Check cmpp4 _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo nomad-client http   passing b4e60280df111aa775ff30a1981cc9d9b7ff7e70 cmpp7],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp7],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Client HTTP Check cmpp4 _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g nomad-client http   passing 688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa cmpp8],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Service 'docker-registry' check cmpp4 docker-registry docker-registry    critical service:docker-registry cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp1],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp3],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp5],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Server HTTP Check cmpp4 _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb nomad http   passing 46143ead15793549f5b34e55e94331c20de1cdec cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [ secure active Vault Sealed Status cmpp4 vault:192.168.1.204:8201 vault    passing vault:192.168.1.204:8201:vault-sealed-check cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Client HTTP Check cmpp4 _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv nomad-client http   passing 95e9bdf85c3c00e9584b4d6104edd6340ce285e8 cmpp6],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp2],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Service 'traefik' check cmpp4 traefik traefik    passing service:traefik cmpp2],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Client HTTP Check cmpp4 _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag nomad-client http   passing 291f56f7f65a736c3c0ee0404f90cfb14c46833a cmpp3],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Server Serf Check cmpp4 _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or nomad  serf  passing 306a2c44f10aa4ff26debfcfa3ef0b23b92f388f cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp8],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [ secure  Service 'traefiktls' check cmpp4 traefiktls traefiktls    passing service:traefiktls cmpp2],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Client HTTP Check cmpp4 _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc nomad-client http   passing d1cd1cbe5327e848371c937f867644ffbef7456a cmpp9],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Nomad Server RPC Check cmpp4 _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes nomad   rpc passing 8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671 secure  Service 'rabbitmqclustertls' check cmpp4 rabbitmqclustertls rabbitmqclustertls    passing service:rabbitmqclustertls cmpp4],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp6],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Service 'elasticsearch' check cmpp4 elasticsearch elasticsearch    passing service:elasticsearch cmpp7],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Service 'sftp' check cmpp4 sftp sftp    passing service:sftp cmpp5],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
mai 20 10:19:13 cmpp4 telegraf[24288]: 2019-05-20T08:19:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [   Serf Health Status cmpp4      passing serfHealth cmpp9],
mai 20 10:19:13 cmpp4 telegraf[24288]: err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"

Expected behavior:

Consul Health Checks are recorded by the prometheus scraping

Actual behavior:

No metrics are stored by prometheus because the prometheus output (/metrics) does not mention consul plugin inputs.

There are closed issues on the same theme with empty tags which were corrected in telegraf 1.10.2 but I still have the same issue.

I tried to tagexclude as described in issue #5522 but I don't really understand the pattern : "Serf SerfHealth ^external ", or with the tag name. But I still have the same issue with telegraf 1.10.4.

For the while I downgraded to 1.9.0 with no output error and prometheus recording metrics.

See also #2422 with older version but reopened.
@danielnelson
Copy link
Contributor

You might need 1.10.3 or later which include #5680.

But based on the error I'm not so sure this is fixed, if 1.10.3 still has errors could you run this command and add the output:

telegraf --input-filter=consul --test

@danielnelson danielnelson added area/prometheus bug unexpected problem or unintended behavior need more info labels May 20, 2019
@osd42
Copy link
Author

osd42 commented May 21, 2019 •
edited
Loading

Hello @danielnelson,

My report was for version 1.10.4, so I compared all versions mentionned in this topic.

extract from 1.9.0 :

[root@cmpp4 telegraf.d]# telegraf --config-directory /etc/telegraf/telegraf.d -input-filter consul -output-filter prometheus_client --debug
2019-05-21T09:24:37Z I! Starting Telegraf 1.9.0
2019-05-21T09:24:37Z I! Using config file: /etc/telegraf/telegraf.conf
2019-05-21T09:24:37Z I! Loaded inputs: inputs.consul
2019-05-21T09:24:37Z I! Loaded aggregators: 
2019-05-21T09:24:37Z I! Loaded processors: 
2019-05-21T09:24:37Z I! Loaded outputs: prometheus_client
2019-05-21T09:24:37Z I! Tags enabled: host=cmpp4
2019-05-21T09:24:37Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"cmpp4", Flush Interval:10s
2019-05-21T09:24:37Z D! [agent] Connecting outputs
2019-05-21T09:24:37Z D! [agent] Attempting connection to output: prometheus_client
2019-05-21T09:24:37Z D! [agent] Successfully connected to output: prometheus_client
2019-05-21T09:24:37Z D! [agent] Starting service inputs
2019-05-21T09:24:50Z D! [outputs.prometheus_client] wrote batch of 25 metrics in 1.15216ms
2019-05-21T09:24:50Z D! [outputs.prometheus_client] buffer fullness: 0 / 10000 metrics.

It works, no errors on prometheus, and metrics are present on the /metrics output.

extract from 1.10.3 :

[root@cmpp4 telegraf.d]# telegraf --config-directory /etc/telegraf/telegraf.d -input-filter consul -output-filter prometheus_client --debug
2019-05-21T09:27:24Z I! Starting Telegraf 1.10.3
2019-05-21T09:27:24Z I! Using config file: /etc/telegraf/telegraf.conf
2019-05-21T09:27:24Z I! Loaded inputs: consul
2019-05-21T09:27:24Z I! Loaded aggregators: 
2019-05-21T09:27:24Z I! Loaded processors: 
2019-05-21T09:27:24Z I! Loaded outputs: prometheus_client
2019-05-21T09:27:24Z I! Tags enabled: host=cmpp4
2019-05-21T09:27:24Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"cmpp4", Flush Interval:10s
2019-05-21T09:27:24Z D! [agent] Connecting outputs
2019-05-21T09:27:24Z D! [agent] Attempting connection to output: prometheus_client
2019-05-21T09:27:24Z D! [agent] Successfully connected to output: prometheus_client
2019-05-21T09:27:24Z D! [agent] Starting service inputs
2019-05-21T09:27:40Z D! [outputs.prometheus_client] wrote batch of 25 metrics in 1.596472ms
2019-05-21T09:27:40Z D! [outputs.prometheus_client] buffer fullness: 0 / 10000 metrics. 
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [secure    service:traefiktls cmpp4 cmpp2 passing   traefiktls Service 'traefiktls' check traefiktls],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ http   291f56f7f65a736c3c0ee0404f90cfb14c46833a cmpp4 cmpp3 passing   nomad-client Nomad Client HTTP Check _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp3 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    306a2c44f10aa4ff26debfcfa3ef0b23b92f388f cmpp4 cmpp4 passing serf  nomad Nomad Server Serf Check _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [  rpc  8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 cmpp4 cmpp4 passing   nomad Nomad Server RPC Check _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    service:sftp cmpp4 cmpp5 passing   sftp Service 'sftp' check sftp],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ http   95e9bdf85c3c00e9584b4d6104edd6340ce285e8 cmpp4 cmpp6 passing   nomad-client Nomad Client HTTP Check _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp1 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp6 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp4 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [secure    service:rabbitmqclustertls cmpp4 cmpp4 passing  external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671 rabbitmqclustertls Service 'rabbitmqclustertls' check rabbitmqclustertls],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [secure   active vault:192.168.1.204:8201:vault-sealed-check cmpp4 cmpp4 passing   vault Vault Sealed Status vault:192.168.1.204:8201],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [secure    service:rabbitmqtls cmpp4 cmpp5 passing   rabbitmqtls Service 'rabbitmqtls' check rabbitmqtls],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    service:traefik cmpp4 cmpp2 passing   traefik Service 'traefik' check traefik],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    service:docker-registry cmpp4 cmpp4 passing   docker-registry Service 'docker-registry' check docker-registry],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ http   688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa cmpp4 cmpp8 passing   nomad-client Nomad Client HTTP Check _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp8 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ http   46143ead15793549f5b34e55e94331c20de1cdec cmpp4 cmpp4 passing   nomad Nomad Server HTTP Check _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp5 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ http   b4e60280df111aa775ff30a1981cc9d9b7ff7e70 cmpp4 cmpp7 passing   nomad-client Nomad Client HTTP Check _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp7 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    service:elasticsearch cmpp4 cmpp7 passing   elasticsearch Service 'elasticsearch' check elasticsearch],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ http   d1cd1cbe5327e848371c937f867644ffbef7456a cmpp4 cmpp9 passing   nomad-client Nomad Client HTTP Check _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp9 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [    serfHealth cmpp4 cmpp2 passing    Serf Health Status ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp1  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  service:traefik cmpp2 traefik Service 'traefik' check passing  cmpp4 traefik   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [serf  306a2c44f10aa4ff26debfcfa3ef0b23b92f388f cmpp4 nomad Nomad Server Serf Check passing  cmpp4 _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  46143ead15793549f5b34e55e94331c20de1cdec cmpp4 nomad Nomad Server HTTP Check passing http cmpp4 _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp5  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  95e9bdf85c3c00e9584b4d6104edd6340ce285e8 cmpp6 nomad-client Nomad Client HTTP Check passing http cmpp4 _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  service:elasticsearch cmpp7 elasticsearch Service 'elasticsearch' check passing  cmpp4 elasticsearch   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  291f56f7f65a736c3c0ee0404f90cfb14c46833a cmpp3 nomad-client Nomad Client HTTP Check passing http cmpp4 _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 cmpp4 nomad Nomad Server RPC Check passing  cmpp4 _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes  rpc ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  service:docker-registry cmpp4 docker-registry Service 'docker-registry' check passing  cmpp4 docker-registry   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  vault:192.168.1.204:8201:vault-sealed-check cmpp4 vault Vault Sealed Status passing  cmpp4 vault:192.168.1.204:8201 secure  active],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp6  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  b4e60280df111aa775ff30a1981cc9d9b7ff7e70 cmpp7 nomad-client Nomad Client HTTP Check passing http cmpp4 _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa cmpp8 nomad-client Nomad Client HTTP Check passing http cmpp4 _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp8  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp3  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp4  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  d1cd1cbe5327e848371c937f867644ffbef7456a cmpp9 nomad-client Nomad Client HTTP Check passing http cmpp4 _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp2  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  service:traefiktls cmpp2 traefiktls Service 'traefiktls' check passing  cmpp4 traefiktls secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [ external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671 service:rabbitmqclustertls cmpp4 rabbitmqclustertls Service 'rabbitmqclustertls' check passing  cmpp4 rabbitmqclustertls secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  service:rabbitmqtls cmpp5 rabbitmqtls Service 'rabbitmqtls' check passing  cmpp4 rabbitmqtls secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  service:sftp cmpp5 sftp Service 'sftp' check passing  cmpp4 sftp   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp7  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [  serfHealth cmpp9  Serf Health Status passing  cmpp4    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:elasticsearch cmpp4 passing    cmpp7 elasticsearch Service 'elasticsearch' check elasticsearch   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [291f56f7f65a736c3c0ee0404f90cfb14c46833a cmpp4 passing    cmpp3 nomad-client Nomad Client HTTP Check _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag  http ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp5  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [95e9bdf85c3c00e9584b4d6104edd6340ce285e8 cmpp4 passing    cmpp6 nomad-client Nomad Client HTTP Check _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv  http ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [b4e60280df111aa775ff30a1981cc9d9b7ff7e70 cmpp4 passing    cmpp7 nomad-client Nomad Client HTTP Check _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo  http ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [d1cd1cbe5327e848371c937f867644ffbef7456a cmpp4 passing    cmpp9 nomad-client Nomad Client HTTP Check _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc  http ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:traefiktls cmpp4 passing    cmpp2 traefiktls Service 'traefiktls' check traefiktls secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [306a2c44f10aa4ff26debfcfa3ef0b23b92f388f cmpp4 passing    cmpp4 nomad Nomad Server Serf Check _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or   serf],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp4  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:rabbitmqclustertls cmpp4 passing  external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671  cmpp4 rabbitmqclustertls Service 'rabbitmqclustertls' check rabbitmqclustertls secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp6  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp7  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp8  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp9  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [46143ead15793549f5b34e55e94331c20de1cdec cmpp4 passing    cmpp4 nomad Nomad Server HTTP Check _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb  http ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 cmpp4 passing rpc   cmpp4 nomad Nomad Server RPC Check _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [vault:192.168.1.204:8201:vault-sealed-check cmpp4 passing   active cmpp4 vault Vault Sealed Status vault:192.168.1.204:8201 secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:sftp cmpp4 passing    cmpp5 sftp Service 'sftp' check sftp   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:docker-registry cmpp4 passing    cmpp4 docker-registry Service 'docker-registry' check docker-registry   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:rabbitmqtls cmpp4 passing    cmpp5 rabbitmqtls Service 'rabbitmqtls' check rabbitmqtls secure  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa cmpp4 passing    cmpp8 nomad-client Nomad Client HTTP Check _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g  http ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp1  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp2  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [service:traefik cmpp4 passing    cmpp2 traefik Service 'traefik' check traefik   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:27:43Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [serfHealth cmpp4 passing    cmpp3  Serf Health Status    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"

extract from 1.10.4 :

[root@cmpp4 telegraf.d]# telegraf --config-directory /etc/telegraf/telegraf.d -input-filter consul -output-filter prometheus_client --debug
2019-05-21T09:28:47Z I! Starting Telegraf 1.10.4
2019-05-21T09:28:47Z I! Using config file: /etc/telegraf/telegraf.conf
2019-05-21T09:28:47Z I! Loaded inputs: consul
2019-05-21T09:28:47Z I! Loaded aggregators: 
2019-05-21T09:28:47Z I! Loaded processors: 
2019-05-21T09:28:47Z I! Loaded outputs: prometheus_client
2019-05-21T09:28:47Z I! Tags enabled: host=cmpp4
2019-05-21T09:28:47Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"cmpp4", Flush Interval:10s
2019-05-21T09:28:47Z D! [agent] Connecting outputs
2019-05-21T09:28:47Z D! [agent] Attempting connection to output: prometheus_client
2019-05-21T09:28:47Z D! [agent] Successfully connected to output: prometheus_client
2019-05-21T09:28:47Z D! [agent] Starting service inputs
2019-05-21T09:29:00Z D! [outputs.prometheus_client] wrote batch of 25 metrics in 1.390948ms
2019-05-21T09:29:00Z D! [outputs.prometheus_client] buffer fullness: 0 / 10000 metrics. 
2019-05-21T09:29:10Z D! [outputs.prometheus_client] wrote batch of 25 metrics in 1.186233ms
2019-05-21T09:29:10Z D! [outputs.prometheus_client] buffer fullness: 0 / 10000 metrics. 
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp9      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp2 traefiktls traefiktls    cmpp4 service:traefiktls Service 'traefiktls' check secure ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp4 nomad _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb http   cmpp4 46143ead15793549f5b34e55e94331c20de1cdec Nomad Server HTTP Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp4 vault vault:192.168.1.204:8201   active cmpp4 vault:192.168.1.204:8201:vault-sealed-check Vault Sealed Status secure ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp4 nomad _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or    cmpp4 306a2c44f10aa4ff26debfcfa3ef0b23b92f388f Nomad Server Serf Check  serf],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [rpc passing cmpp4 nomad _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes    cmpp4 8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 Nomad Server RPC Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp4      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp4 rabbitmqclustertls rabbitmqclustertls  external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671  cmpp4 service:rabbitmqclustertls Service 'rabbitmqclustertls' check secure ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp5      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp2 traefik traefik    cmpp4 service:traefik Service 'traefik' check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp3 nomad-client _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag http   cmpp4 291f56f7f65a736c3c0ee0404f90cfb14c46833a Nomad Client HTTP Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp3      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp6      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp7      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp8 nomad-client _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g http   cmpp4 688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa Nomad Client HTTP Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp7 elasticsearch elasticsearch    cmpp4 service:elasticsearch Service 'elasticsearch' check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp8      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp1      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp5 rabbitmqtls rabbitmqtls    cmpp4 service:rabbitmqtls Service 'rabbitmqtls' check secure ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp7 nomad-client _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo http   cmpp4 b4e60280df111aa775ff30a1981cc9d9b7ff7e70 Nomad Client HTTP Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp6 nomad-client _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv http   cmpp4 95e9bdf85c3c00e9584b4d6104edd6340ce285e8 Nomad Client HTTP Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp9 nomad-client _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc http   cmpp4 d1cd1cbe5327e848371c937f867644ffbef7456a Nomad Client HTTP Check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp2      cmpp4 serfHealth Serf Health Status  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp4 docker-registry docker-registry    cmpp4 service:docker-registry Service 'docker-registry' check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_passing, labels: [ passing cmpp5 sftp sftp    cmpp4 service:sftp Service 'sftp' check  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_passing"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4 nomad 46143ead15793549f5b34e55e94331c20de1cdec Nomad Server HTTP Check _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb   cmpp4 passing  http  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp6  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp7 nomad-client b4e60280df111aa775ff30a1981cc9d9b7ff7e70 Nomad Client HTTP Check _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo   cmpp4 passing  http  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp8 nomad-client 688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa Nomad Client HTTP Check _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g   cmpp4 passing  http  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp6 nomad-client 95e9bdf85c3c00e9584b4d6104edd6340ce285e8 Nomad Client HTTP Check _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv   cmpp4 passing  http  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp7  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp7 elasticsearch service:elasticsearch Service 'elasticsearch' check elasticsearch   cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp8  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp2  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4 nomad 306a2c44f10aa4ff26debfcfa3ef0b23b92f388f Nomad Server Serf Check _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or   cmpp4 passing   serf ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4 vault vault:192.168.1.204:8201:vault-sealed-check Vault Sealed Status vault:192.168.1.204:8201  active cmpp4 passing secure   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp5 rabbitmqtls service:rabbitmqtls Service 'rabbitmqtls' check rabbitmqtls   cmpp4 passing secure   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp9 nomad-client d1cd1cbe5327e848371c937f867644ffbef7456a Nomad Client HTTP Check _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc   cmpp4 passing  http  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp9  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp5 sftp service:sftp Service 'sftp' check sftp   cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp2 traefik service:traefik Service 'traefik' check traefik   cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp3 nomad-client 291f56f7f65a736c3c0ee0404f90cfb14c46833a Nomad Client HTTP Check _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag   cmpp4 passing  http  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp3  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4 nomad 8c71ffc9a5a3c5329dafe9350ce60d6761e102a0 Nomad Server RPC Check _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes   cmpp4 passing    rpc],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4 rabbitmqclustertls service:rabbitmqclustertls Service 'rabbitmqclustertls' check rabbitmqclustertls external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671  cmpp4 passing secure   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp5  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp1  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp2 traefiktls service:traefiktls Service 'traefiktls' check traefiktls   cmpp4 passing secure   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4  serfHealth Serf Health Status    cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_critical, labels: [cmpp4 docker-registry service:docker-registry Service 'docker-registry' check docker-registry   cmpp4 passing    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_critical"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Service 'docker-registry' check docker-registry   passing docker-registry service:docker-registry    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp5 Service 'rabbitmqtls' check rabbitmqtls   passing rabbitmqtls service:rabbitmqtls secure   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp7 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp1 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp3 Nomad Client HTTP Check _nomad-client-ls5s2bgyo4jg2m6xdaig25ymx7gyusag http  passing nomad-client 291f56f7f65a736c3c0ee0404f90cfb14c46833a    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp3 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp9 Nomad Client HTTP Check _nomad-client-yby4ot6g4kwrogz6hc6ji7spdms3rkqc http  passing nomad-client d1cd1cbe5327e848371c937f867644ffbef7456a    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Nomad Server HTTP Check _nomad-server-bxlytywgpy2xllgrvmpcqn5t3oxlroqb http  passing nomad 46143ead15793549f5b34e55e94331c20de1cdec    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp7 Nomad Client HTTP Check _nomad-client-rjbeenpvdntwwspcpwu7tlc2gp5zudwo http  passing nomad-client b4e60280df111aa775ff30a1981cc9d9b7ff7e70    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp7 Service 'elasticsearch' check elasticsearch   passing elasticsearch service:elasticsearch    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Service 'rabbitmqclustertls' check rabbitmqclustertls   passing rabbitmqclustertls service:rabbitmqclustertls secure  external.entrypoint=amqps://rabbitmqclustertls.service.proxmox.sims:5671 ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Vault Sealed Status vault:192.168.1.204:8201   passing vault vault:192.168.1.204:8201:vault-sealed-check secure   active],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp8 Nomad Client HTTP Check _nomad-client-gb6wf2qmafgedf4tifuwmmhouyyzpk3g http  passing nomad-client 688eacc3c385e9c67e64c0c9b20f8ccc1e9415aa    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp2 Service 'traefiktls' check traefiktls   passing traefiktls service:traefiktls secure   ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Nomad Server Serf Check _nomad-server-pbtqaum7rmjordy76t5gtbogn3euj4or   passing nomad 306a2c44f10aa4ff26debfcfa3ef0b23b92f388f  serf  ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp4 Nomad Server RPC Check _nomad-server-h7v2dgembqfgxkcxqs6qtsafgcvfyjes  rpc passing nomad 8c71ffc9a5a3c5329dafe9350ce60d6761e102a0    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp5 Service 'sftp' check sftp   passing sftp service:sftp    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp6 Nomad Client HTTP Check _nomad-client-rkftgbhtjlo53ioaob7tbcv7w3qqpqtv http  passing nomad-client 95e9bdf85c3c00e9584b4d6104edd6340ce285e8    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp6 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp8 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp9 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp2 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp2 Service 'traefik' check traefik   passing traefik service:traefik    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"
2019-05-21T09:29:13Z E! Error creating prometheus metric, key: consul_health_checks_warning, labels: [cmpp4 cmpp5 Serf Health Status    passing  serfHealth    ],
err: "external_entrypoint_amqps:__rabbitmqclustertls_service_proxmox_sims:5671" is not a valid label name for metric "consul_health_checks_warning"

Best regards

@osd42
Copy link
Author

osd42 commented May 27, 2019 •
edited
Loading

Hello,
Is my report relevant ? Do you need anything else ?

For now, I decided not to put consul input in production because the latest procstat is simply awesome on 1.10.x. I'll retry consul_exporter later, but I am available to test anything you need on consul input.

Best regards,

Charles, OSD42

@osd42
Copy link
Author

osd42 commented Jul 30, 2019

Hello,
I tried with telegraf 1.11.3 and it is working. I wish to thank you for the support !

telegraf --config-directory /etc/telegraf/telegraf.d -input-filter consul -output-filter prometheus_client --debug

output :
2019-07-30T14:05:48Z I! Loaded processors: 2019-07-30T14:05:48Z I! Loaded outputs: prometheus_client 2019-07-30T14:05:48Z I! Tags enabled: host=pp221-nomad-consul 2019-07-30T14:05:48Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"pp221-nomad-consul", Flush Interval:10s 2019-07-30T14:05:48Z D! [agent] Connecting outputs 2019-07-30T14:05:48Z D! [agent] Attempting connection to output: prometheus_client 2019-07-30T14:05:48Z D! [agent] Successfully connected to output: prometheus_client 2019-07-30T14:05:48Z D! [agent] Starting service inputs 2019-07-30T14:06:00Z D! [outputs.prometheus_client] wrote batch of 15 metrics in 556.662µs 2019-07-30T14:06:00Z D! [outputs.prometheus_client] buffer fullness: 0 / 10000 metrics.

Have a good day !

@osd42 osd42 closed this as completed Jul 30, 2019
@powersj powersj mentioned this issue Sep 13, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/prometheus bug unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@danielnelson @osd42