CompactionE2EIdempotencyTest.test_basic_compaction: last offset off by one error

[rystsov]

Sometimes when a consumer queries last offset (least upper bound)

consumer.seekToEnd(tps);
long lub = consumer.position(tp);

and reads until it reaches it

consumer.seekToBeginning(tps);
while (consumer.position(tp) < lub) {
    consumer.poll(...);
}

it gets stuck because the position stops advancing.

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/22632#01862925-396e-479f-9ea0-86aaa7b2cc52

In this example (see initial_cleanup_policy=compact.workload=Workload.TX_UNIQUE_KEYS/7/) we see that the position of consumers reached LUB (30120, 30541); but it didn't happen with the stuck consumer (36979 < 36980). The problem is with position and not with the data: all consumers read all written data (36978, 30112, 30535).

[DEBUG - 2023-02-07 00:01:11,420 - compacted_verifier - _remote_info - lineno:245]: Received {
    "total_writes": 74980,
    "total_reads": 74980,
    "min_writes": 23010,
    "min_reads": 23010,
    "partitions": [
        {
            "partition": 0, 
            "end_offset": 36980,
            "read_offset": 36978,
            "read_position": 36979,
            "written_offset": 36978,
            "consumed": false
        },
        {
            "partition": 1,
            "end_offset": 30120,
            "read_offset": 30112,
            "read_position": 30120,
            "written_offset": 30112,
            "consumed": true
        },
        {
            "partition": 2,
            "end_offset": 30541,
            "read_offset": 30535,
            "read_position": 30541,
            "written_offset": 30535,
            "consumed": true
        }
    ]
}

The issue was found with the compaction_e2e_test test which:

1. writes data
2. waits until it's compacted
3. reads data

What's interesting is that each consumer reads its own partition and the partition of the failed consumer experienced re-election after the first step.

[rystsov]

the test is disable by relaxation of the invariant (all written records are consumed vs it's possible to consume up to an offset returned by list offsets) instead of adding ok_to_fail, see 3e4a479

[rystsov]

Might be related:

• https://redpandacommunity.slack.com/archives/C01B3QBLUTT/p1694094516786129
• https://redpandadata.slack.com/archives/C01ND4SVB6Z/p1694098547528039