Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Handle Kubernetes API conflicts properly for signing keys #1835

Merged

Conversation

nabokihms
Copy link
Member

Running two dex instances in HA mode in the Kubernetes cluster, we are constantly receiving messages like this from one instance:

{"level":"info","msg":"keys rotated, next rotation: 2020-10-08 15:33:14.700531412 +0000 UTC","time":"2020-10-08T09:33:14Z"}

And errors from another instance:

{"level":"error","msg":"failed to rotate keys: PUT https://*****/apis/dex.coreos.com/v1/namespaces/******/signingkeies/openid-connect-keys Conflict: response from server \"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Operation cannot be fulfilled on signingkeies.dex.coreos.com \\\"openid-connect-keys\\\": the object has been modified; please apply your changes to the latest version and try again\",\"reason\":\"Conflict\",\"details\":{\"name\":\"openid-connect-keys\",\"group\":\"dex.coreos.com\",\"kind\":\"signingkeies\"},\"code\":409}\"","time":"2020-10-09T09:33:16Z"}

This happens because both instances get the same next rotation time from a single openid-connect-keys object and start rotating synchronously. In the case of using Kubernetes storage, we have to treat the conflict response code from Kubernetes API as a result of the work of the second instance.

P.S. Error errAlreadyRotated located on the server level. I tried to avoid mixing it with the storage code. Need to figure out the right way to propagate errors from the storage level to the server level (but it does not influence the main idea of this PR).

@nabokihms nabokihms force-pushed the kubernetes-client-keys-conflicts-fix branch from ee9f5c2 to f33c5de Compare October 12, 2020 05:59
Signed-off-by: m.nabokikh <maksim.nabokikh@flant.com>
@nabokihms nabokihms force-pushed the kubernetes-client-keys-conflicts-fix branch from f33c5de to 4801b2c Compare October 12, 2020 06:15
Copy link
Member

@bonifaido bonifaido left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @nabokihms !

@sagikazarmark sagikazarmark added this to the v2.26.0 milestone Oct 13, 2020
@bonifaido bonifaido merged commit 28b2350 into dexidp:master Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants