Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.1.x] cloud_storage: Prevent segment reuploads from adjacent segment merger #9845

Merged
merged 4 commits into from
Apr 5, 2023
Merged

[v23.1.x] cloud_storage: Prevent segment reuploads from adjacent segment merger #9845

merged 4 commits into from
Apr 5, 2023

Commits on Apr 5, 2023

  1. cloud_storage: Improve logging in adjacent segment merger 

    (cherry picked from commit 3c7cbb9)
    @Lazin
    Lazin committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    0d06079 View commit details
    Browse the repository at this point in the history

  2. cloud_storage: Prevent reuploads with unexpected size 

    It is possible that the reuploaded segment is smaller than predicted. In
this case it's not safe to reupload it because adjacent segment merging
may try to reupload it second time. This commit blocks reupload if the
size doesn't match the result of the manifest scan.

(cherry picked from commit dc855e2)
    @Lazin
    Lazin committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    2f85d5f View commit details
    Browse the repository at this point in the history

  3. cloud_storage: Update segment_collector 

    Add new optional parameter to the c-tor. The parameter accepts inclusive
end offset of the offset range. The segment_collector is forced to reach
the target end offset if it's set. This allows adjacent_segment_merger
to reupload offset ranges more precisely since it's no longer allowed to
have some slack.

(cherry picked from commit 441fb14)
    @Lazin
    Lazin committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    acb854f View commit details
    Browse the repository at this point in the history

  4. cloud_storage: Fix race condition in the segment_collector 

    The segment_collector calculates the size of the segment sequence during
the initial scan and then it uses it to produce an upload candidate. The
'make_upload_candidate' method is a future with several scheduling
points. If the size of the last segment changes during the call the
method will return incorrect content length. This commit fixes this by
caching the size of the last segment.

The problem only affect adjacent segment reuploads. It shouldn't affect
compacted segment reupload because the last compacted segment is always
rolled and its size can't change.

(cherry picked from commit ccb0e0a)
    @Lazin
    Lazin committed Apr 5, 2023
    Configuration menu
    Copy the full SHA
    6dc9dab View commit details
    Browse the repository at this point in the history