Fix Query Frontend grpc settings to avoid noisy error log

[annanay25]

Signed-off-by: Annanay annanayagarwal@gmail.com

What this PR does:
Fix Query Frontend GRPC settings to fix the noisy error message

caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"

Thanks to @pstibrany for digging into this in Cortex - cortexproject/cortex#3606 (comment)

Which issue(s) this PR fixes:
Fixes #493

Checklist

• NA Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Oh dang, nice!

Does this make grpc heartbeat more frequently between querier->queryfrontend so the disconnects don't happen? if so can we default to these lower values?

[mdisibio]

Do we want to set defaults in .go, rather than require most users to set these settings to avoid noisy errors?

[annanay25]

Thanks for the comments. Updated the PR to make the changes in the .go file

@joe-elliott - these settings will not affect the heartbeat messages frequency. It is a setting on the server that prevents it from sending a GOAWAY message if a client sends heartbeat messages frequently (due to lack of real traffic)

[pstibrany]

Here is one way of changing the default, without requiring change in the upstream library: cortexproject/cortex#4168.