Add retry middleware in query-frontend

[kvrhdn]

What this PR does:
Requests from the query-frontend to the querier might fail when a querier is starting or stopping. Simply retrying this request usually solves this.

This PR adds a retry middleware which will relaunch requests (up to 5 times by default) if the request failed (http 500 or error).

Which issue(s) this PR fixes:
Related to #761

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[kvrhdn]

This PR also fixes context propagation in the other middleware functions.

Before and after:

[mapno]

Changes look good.

My concern is if this change could lead to hammering down a slow or overloaded querier, making the issue worse. It might be worth considering using a backoff for the retries or other mitigation alternatives.

[kvrhdn]

My concern is if this change could lead to hammering down a slow or overloaded querier, making the issue worse. It might be worth considering using a backoff for the retries or other mitigation alternatives.

In Cortex the querier pulls work from the queue in query-frontend. The amount of queries that are processed concurrently by a querier is limited by MaxConcurrentQueries.
Retrying a request will put this request on the queue again so another querier can pick this up.

BTW, this PR is pretty much the same code as Cortex uses in their queryrange processor which shards requests by day.
https://github.com/cortexproject/cortex/blob/master/pkg/querier/queryrange/retry.go

[joe-elliott]

one nit. nice addition.

[joe-elliott]

continue?

does this do anything?

[kvrhdn]

Well, it continues the for loop 😛

I've removed it, I refactored my loop a couple of times for clarity.

[mapno]

My concern is if this change could lead to hammering down a slow or overloaded querier, making the issue worse. It might be worth considering using a backoff for the retries or other mitigation alternatives.

In Cortex the querier pulls work from the queue in query-frontend. The amount of queries that are processed concurrently by a querier is limited by MaxConcurrentQueries.
Retrying a request will put this request on the queue again so another querier can pick this up.

BTW, this PR is pretty much the same code as Cortex uses in their queryrange processor which shards requests by day.
https://github.com/cortexproject/cortex/blob/master/pkg/querier/queryrange/retry.go

Thanks for the explanation