New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cloud_storage: make cluster default read/write properties "sticky" #6917

Closed

jcsp opened this issue Oct 25, 2022 · 1 comment · Fixed by #6950

Assignees

Labels

area/cloud-storage kind/enhance

Contributor

jcsp commented Oct 25, 2022

This is a step in the direction of #6627

Currently, we provide global cluster defaults for tiered storage, which apply continuously to any topic that does not have an explicit setting for remote.read/write. This creates strange situations if this cluster default is changed: we might switch off tiered storage for an existing topic, while leaving its data behind in S3, or toggle if off/on and leave gaps in S3. This problem still exists when amending individual topics, but that is less impactful than potentially doing this to every topic on the system with one seemingly innocent config edit.

Instead, we should treat these defaults as "sticky" to topics: i.e the topic's tiered storage mode will be whatever the cluster defaults said at the time it was created, or at the time the cluster was upgraded to 22.3.

Ultimately changing the tiered storage state on a topic should be a higher level operation (perhaps admin API driven) that provides proper options for "switch off tiered storage and delete data" or "switch off tiered storage temporarily, and don't evict local data until it's back on" modes, a la #6627.

jcsp added kind/enhance area/cloud-storage labels

jcsp mentioned this issue

cloud_storage: migrate legacy topic configs on upgrade #6876

Merged

6 tasks

Contributor

VladLazar commented Oct 25, 2022

I've got a draft PR for this: #6713. The upgrade handling should be changed to use the migration mechanism from #6876.

jcsp added a commit to jcsp/redpanda that referenced this issue


          features/migrators: make SI mode sticky on upgrade to 22.3

6890cd4

Fixes: redpanda-data#6917

jcsp mentioned this issue

cloud_storage: make cluster default read/write properties "sticky" #6950

Merged

6 tasks

jcsp added a commit to jcsp/redpanda that referenced this issue


          features/migrators: make SI mode sticky on upgrade to 22.3

5a5b279

Fixes: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          tests: update test_shadow_indexing_mode_persistence

73ae5af

...for sticky tiered storage configuration: the defaults
at cluster level now only apply at topic creation time,
and remain set on that topic thereafter.

Related: redpanda-data#6917

jcsp self-assigned this

jcsp added a commit to jcsp/redpanda that referenced this issue


          features/migrators: make SI mode sticky on upgrade to 22.3

33d01fe

Fixes: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          tests: update test_shadow_indexing_mode_persistence

94647fc

...for sticky tiered storage configuration: the defaults
at cluster level now only apply at topic creation time,
and remain set on that topic thereafter.

Related: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          features/migrators: make SI mode sticky on upgrade to 22.3

0045dd9

Fixes: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          tests: update test_shadow_indexing_mode_persistence

3c11c2b

...for sticky tiered storage configuration: the defaults
at cluster level now only apply at topic creation time,
and remain set on that topic thereafter.

Related: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          features/migrators: make SI mode sticky on upgrade to 22.3

4ec4bdd

Fixes: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          tests: update test_shadow_indexing_mode_persistence

8b4c2cf

...for sticky tiered storage configuration: the defaults
at cluster level now only apply at topic creation time,
and remain set on that topic thereafter.

Related: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          features/migrators: make SI mode sticky on upgrade to 22.3

3dd4966

Fixes: redpanda-data#6917

jcsp added a commit to jcsp/redpanda that referenced this issue


          tests: update test_shadow_indexing_mode_persistence

26961be

...for sticky tiered storage configuration: the defaults
at cluster level now only apply at topic creation time,
and remain set on that topic thereafter.

Related: redpanda-data#6917

jcsp closed this as completed in #6950

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment