ISSUE-5969: Acked messages unexpectedly redelivered when others are negatively acked

[sijie]

Original Issue: apache#5969

---

Describe the bug

We've encountered an issue in which acknowledged messages are redelivered one or more times after other messages are negatively acknowledged. This seems to occur when messages are produced in batches. This happens in the absence of any known broker or connection failures.

To Reproduce

I've modified the NegativeAcksTest to test for the correct behavior here: apache/pulsar@master...gmethvin:negative-ack-duplicates

As the test demonstrates, in some configurations positively acknowledged messages are redelivered. This is similar to a situation we see in production.

Expected behavior

Only the negatively acknowledged messages should be redelivered. Positively acknowledged messages should not be redelivered, at least not in a typical situation with no failures.

We produce messages in batches, but both the APIs and the documentation suggest that both acks and negative acks act on a per-message level. If negative acks act on batches, then the APIs and documentation should be changed to clearly indicate that.