Roll segments early under disk pressure, for topics with retention settings

[jcsp]

Problem

Consider a topic with 100 partitions, retention.bytes set to 100MB, the default 1GB segment size limit, and a 50GB drive.

We would have to write 100GB of data (well beyond the drive capacity) before any segments roll, and retention rules are only applied to sealed segments.

The result is that we would hit our threshold for disk-nearly-full and start blocking writes.

Solution

When applying retention rules to a topic, we should notice if the system is under disk pressure, and if we have an opportunity to free some space by applying retention rules, then prompt the log to roll now, even if it has not yet reached the configured segment.bytes.

Impact

Users who have configured appropriate retention.bytes settings for their partition count will no longer run the risk of filling up their disk.

JIRA Link: CORE-1019