Parquet page caching

[joe-elliott]

What this PR does:
Builds on #3166 to add parquet "page" level caching. Since reads from the parquet library are opaque we don't actually attempt to determine whether or not a given read is a page. Instead we simply use the read buffer size which controls how much data is pulled at once from object storage.

Other fixes/changes:

• Added a "none" cache role to allow reads to signal no caching
• Correctly add defaults for background cache to the cache provider. This was missed in the last PR and causing a panic when using cache.
• Fixed a bug that was preventing footer caching from actually doing anything.

Points of discussion:

• This changes blurs the lines between "caching policy" and "role" here. It's not clear to me whether or not the buffer size check/determination if the read should be cached should be made here or in the cache layer.
• Removed the buffered reader from the search path.
• Pages must be copied before being placed into the cache queue. The write queue is async and it's possible for parquet to reuse the buffer before it's written resulting in corrupted data in cache.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[knylander-grafana]

@joe-elliott Will we need doc for this? I can create a doc issue, if so.

[joe-elliott]

@knylander-grafana good call. in this PR i'll add the existence of the new settings option.

for the next release (or before) we should consider the following docs regarding the caching changes:

• update the search perf tuning page to include the new cache options and how they can be used
• migration docs from the old cache settings to the new

[zalegrala]

This looks pretty good to me. Nice work.

[stoewer]

LGTM

[stoewer]

I'm not sure if I understand this correctly: is this necessary because p.caches contain the same cache multiple times?

[joe-elliott]

Yup. If a cache has multiple roles we add it to the map multiple times here:

tempo/modules/cache/cache.go

Lines 60 to 62 in 8d0ddfd

	for _, role := range cacheCfg.Role {
	p.caches[role] = c
	}

So this is just a bit of bookkeeping to prevent stopping one twice.