Add retries to copying files in GCS

[mapno]

What this PR does:

Adds retries to idempotent write operations (non-hedged). Also better propagates errors failing to mark blocks as compacted.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

Comparing this to #2108: This is the safer option and support adding this to the codebase. I'm also good on #2108 but it would require more detailed consideration than this change. i.e. how does #2108 interact with hedging? or impact query speeds?

Ideally a full "fix" would take into account the different needs of the compactor and the querier. The compactor works best with less aggressive and more consistent settings. On the other hand, we'd prefer the querier to very aggressively hit the backend and fail fast if there was an issue so the job could be retried.

Also, ideally we should fix the issue where the compaction is considered successful before the meta rename.

[zalegrala]

I agree we should take this one over #2108, and that we would want to probably make hedging mutually exclusive with retries, and so some other work is needed there.