Statsd: Allow configuring custom tag for instance identification

[joekohlsdorf]

Description

Add optional configuration option to StatsD metrics which allows an operator to configure a custom tag which is sent with all metrics to allow instance identification when running multiple Airflow instances.

Use case/motivation

When running multiple Airflow instances on the same host/K8s cluster, there is no good way to filter StatsD metrics by instance.
The only option which currently exists is to modify the statsd_prefix which however makes it impossible to reuse the same dashboards / alerts for several Airflow instances in monitoring tools like Datadog.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[hussein-awala]

Indeed this is available now only for datadog metrics via the config statsd_datadog_tags, where for the other monitoring tools, we are using statsd.StatsClient or any custom client which extend it, and unfortunately, this client doesn't support tagging (doc)

Tagged metrics—such as those used by Datadog and Telegraf—are explicitly outside the scope of this library. Alternatives exist and are recommended. This document lays out the reasons to avoid support for tags.

I'm testing alternative solutions to support tagging on all the backends in order to generalize #28961, and this will include what you are proposing. If you have any idea to do that, feel free to propose it or open a new PR.

[joekohlsdorf]

Already available for Datadog and not applicable for other systems.