Fix perf when streams don't change often

[erikjohnston]

There is a bug with the StreamChangeCache where it would incorrectly return that all entities had changed if asked for entities changed since the earliest stream position.

Note that for streams we use the inequalities: $min_stream_id < stream_id <= $max_stream_id, i.e. when we ask the stream change cache for all things that have changed since $stream_id we don't care for events that happened at $stream_id.

Specifically: _earliest_known_stream_pos is the position at which we know that we'll have entries for all changes since that point, we can use the cache for any stream IDs that equal _earliest_known_stream_pos.

_earliest_known_stream_pos is set in three places:

• On startup we set it either to:
  • the current maximum stream ID, with not prefilled values; or
  • the minimum of the latest N values we pulled from the DB
• When we evict items from the bottom, we set it to the stream ID of the evicted items.

This was changed in matrix-org/synapse#14435, but I think we were overly conservative there.

[clokep]

I believe this looks correct, I think it is missing 1 tweak to the tests though.

[clokep]

We had a long conversation about this @ matrix-org/synapse#14435 (comment), during which I said:

I'm a bit skeptical of the resulting consistency between strict and non-strict comparisons, but would like a double checking.

Your response was:

as if we call entity_has_changed for a stream pos then has_entity_changed must return True for that stream pos. Either because we remember the entity has changed or because the stream pos is before the minimum threshold

It looks like I interpreted "before" incorrectly in that case, since at the stream position we do have the "proper" information needed to calculate whether there were changes or not.

The comment for _earliest_known_stream_pos even makes this clear:

synapse/synapse/util/caches/stream_change_cache.py

Lines 91 to 93 in ef9ef99

	# the earliest stream_pos for which we can reliably answer
	# get_all_entities_changed. In other words, one less than the earliest
	# stream_pos for which we know _cache is valid.

tl;dr Before #14435 we were inconsistently checking if it was strictly before or before-or-equal, we made them consistent in that PR, but to the wrong one.

[clokep]

Should we add checks for 3 (near line 35)?