Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase unseal retry limit #1605

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

alemorcuq
Copy link
Collaborator

Description of the change
Increase the number of maximum unseal retries to 15.

The original first 5 retries happen in just 200ms, which is not enough time for some scenarios, such as the one described in #1599.
Increasing this number will ensure the exponential backoff takes off.

Here's an excerpt from the logs that shows the time at which each retry happens (starting at 1, since 0 shows the first attempt which is not a retry):

$ kubectl logs -n kube-system deploy/sealed-secrets-controller | grep Updating | nl -v 0
     0	time=2024-09-29T11:49:35.764Z level=INFO msg=Updating key=default/my-secret
     1	time=2024-09-29T11:49:35.777Z level=INFO msg=Updating key=default/my-secret
     2	time=2024-09-29T11:49:35.794Z level=INFO msg=Updating key=default/my-secret
     3	time=2024-09-29T11:49:35.822Z level=INFO msg=Updating key=default/my-secret
     4	time=2024-09-29T11:49:35.867Z level=INFO msg=Updating key=default/my-secret
     5	time=2024-09-29T11:49:35.955Z level=INFO msg=Updating key=default/my-secret
     6	time=2024-09-29T11:49:36.123Z level=INFO msg=Updating key=default/my-secret
     7	time=2024-09-29T11:49:36.451Z level=INFO msg=Updating key=default/my-secret
     8	time=2024-09-29T11:49:37.098Z level=INFO msg=Updating key=default/my-secret
     9	time=2024-09-29T11:49:38.389Z level=INFO msg=Updating key=default/my-secret
    10	time=2024-09-29T11:49:40.957Z level=INFO msg=Updating key=default/my-secret
    11	time=2024-09-29T11:49:46.088Z level=INFO msg=Updating key=default/my-secret
    12	time=2024-09-29T11:49:56.338Z level=INFO msg=Updating key=default/my-secret
    13	time=2024-09-29T11:50:16.823Z level=INFO msg=Updating key=default/my-secret
    14	time=2024-09-29T11:50:57.793Z level=INFO msg=Updating key=default/my-secret
    15	time=2024-09-29T11:52:19.723Z level=INFO msg=Updating key=default/my-secret
    16	time=2024-09-29T11:55:03.572Z level=INFO msg=Updating key=default/my-secret

Benefits
The controller takes longer to give up on a Sealed Secret.

Possible drawbacks
There will be more retries when the unseal error is legit, but that shouldn't cause any issues because of the exponential backoff.

Applicable issues

Signed-off-by: Alejandro Moreno <alemorcuq@gmail.com>
@agarcia-oss
Copy link
Member

Hi @alemorcuq, I wonder if it'll be a better option to make the number of retries a configurable flag for the controller instead. WDYT?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Secret deleted by the garbage collector with delay
2 participants