kube_service_labels metric has inconsistent label dimensions

[cofyc]

Hi, guys

In recent commit 1f8761d, prometheus client_go library is updated, it checks inconsistent label dimensions now, /metrics will report following errors (test against kubernetes ./hack/local-up-cluster.sh):

# curl -s http://localhost:8081/metrics
An error has occurred during metrics gathering:

collected metric kube_service_labels label:<name:"label_addonmanager_kubernetes_io_mode" value:"Reconcile" > label:<name:"label_k8s_app" value:"kube-dns" > label:<name:"label_kubernetes_io_cluster_service" value:"true" > label:<name:"label_kubernetes_io_name" value:"KubeDNS" > label:<name:"namespace" value:"kube-system" > label:<name:"service" value:"kube-dns" > gauge:<value:1 >  has label dimensions inconsistent with previously collected metrics in the same metric family

I checked the previous version of prometheus client_go library, found that checkMetricConsistency function (which is used to check inconsistent label) is disabled (b12f83d).

Should we change kube_service_labels metric to have consistent label dimension or update current prometheus client_go library to disable checkMetricConsistency functionality?

[matthiasr]

I think for this particular metric we need to disable it. Labels are necessarily inconsistent between Kubernetes objects, even of the same kind. Is it possible to disable this check just for the _labels metrics?

[matthiasr]

As a hotfix I think we should disable it wholesale to unbreak things.

[brancz]

Yes this was my bad, I should have caught it when reviewing. We previously disabled this check manually. I'll get right to it. Thanks a lot for reporting!