Known Issue with label_replace Queries in Prometheus Versions 2.51 and 2.52

[rahulguptajss]

There is a known issue in Prometheus versions 2.51 and 2.52, which is relevant if you are utilizing label_replace queries. Here's an example to illustrate the problem:

Suppose an aggregate move is performed on a volume. If you subsequently execute a query like the one below, you might encounter an issue execution: vector cannot contain metrics with the same labelset

label_replace(volume_avg_latency{
  datacenter=~"$Datacenter",
  cluster=~"$Cluster",
  svm=~"$SVM",
  volume=~"$Volume",
  style!="flexgroup_constituent"
},"aggr", "", "aggr", ".*")   
and 
topk(
  $TopResources, 
  avg_over_time(
    label_replace(volume_avg_latency{
  datacenter=~"$Datacenter",
  cluster=~"$Cluster",
  svm=~"$SVM",
  volume=~"$Volume",
  style!="flexgroup_constituent"
},"aggr", "", "aggr", ".*")[3h:] @ end()
  )
)

This query may return the following error:

Unfortunately, there is no known workaround for this issue at the moment. You can refer to the Prometheus bug report at prometheus/prometheus#13837 for more details. Although it appears to have been fixed in Prometheus 2.52, the problem persists.

Here's an example of the issue:

Discord conversation here