Add k8s service discovery resource attributes

[dashpole]

Description:

I've found myself writing relabel rules to take labels from prometheus service discovery and then wondering how to convert from metric labels to resource labels. In GKE, we even wrote a custom processor to do this for us. I realized the other day that we actually have all of these labels available to us in the prometheus receiver.

This PR converts some of the service discovery __meta labels from prometheus' kubernetes service discovery into otel semantic conventions. This could be extended to include other service discovery implementations (e.g. gce, aws, azure, docker swarm, etc.), or we could even add configuration to allow users to define their own map for prom service discovery labels to prometheus resource attributes.

This is most useful when specifying - role: pod (Used by the PodMonitor prom-operator resource) or - role: node in kubernetes_sd_configs, as OpenTelemetry k8s attributes map well to the set of prometheus service discovery labels. In particular, - role: endpoints (used by ServiceMonitor resource) and -role: service (used by Probe resource) doesn't add any particularly useful attributes.

Example

With this collector config:

receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: $$1:$$2
          target_label: __address__
exporters:
  logging:
    logLevel: debug
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [logging]

You get the following resource labels in the output:

DEBUG	loggingexporter/logging_exporter.go:66	ResourceMetrics #0
Resource SchemaURL:
Resource labels:
     -> service.name: STRING(kubernetes-pods)
     -> net.host.name: STRING(xx.xx.x.x)
     -> service.instance.id: STRING(xx.xx.x.x:80)
     -> net.host.port: STRING(80)
     -> http.scheme: STRING(http)
     -> k8s.pod.name: STRING(kube-dns-697dc8fc8b-zl95s)
     -> k8s.pod.uid: STRING(adc8575a-2d85-49bb-907b-16626182ec3d)
     -> k8s.container.name: STRING(prometheus-to-sd)
     -> k8s.node.name: STRING(gke-dashpole-test-default-pool-61f071a2-1re0)
     -> k8s.replicaset.name: STRING(kube-dns-697dc8fc8b)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope
Metric #0
Descriptor:
     -> Name: up
     -> Description: The scraping was successful
     -> Unit:
     -> DataType: Gauge

[dashpole]

cc @open-telemetry/wg-prometheus

[jpkrohling]

Changelog conflict resolved.