[release-1.25] Failing secret webhook causes unexpected behavior

[brandond]

• Failing secret webhook causes unexpected behavior #7654 backport

[ShylajaDevadiga]

Validated using commit id on release-1.25 branch using commit id 873fc1c

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

Cluster Configuration:
Two node, One server, One agent

Steps to reproduce as shared in the issue

1. Install k3s
2. Create a bad webhook configuration (this one just calls out to httpbin.org for a 502 error response)

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: rancher.cattle.io
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    url: https://httpbin.org/status/502
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rancher.cattle.io.secrets
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - secrets
    scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 5

3. Attempt to join a new agent

Results from reproducing the issue

Agent is unable to join the cluster. Hangs during the joining process.

$ kubectl get nodes
NAME             STATUS   ROLES                  AGE   VERSION
ip-172-31-9-76   Ready    control-plane,master   14m   v1.25.10+k3s1

Logs from server node:

Jun 15 18:05:22 ip-172-31-9-76 k3s[2161]: time="2023-06-15T18:05:22Z" level=error msg="Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://httpbin.org/status/502?timeout=5s\": context deadline exceeded"

Logs from agent node:

Jun 15 18:03:18 ip-172-31-4-215 k3s[2140]: time="2023-06-15T18:03:18Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"

Results from validating the issue:

Agent has joined the cluster.

$ kubectl get nodes
NAME               STATUS   ROLES                  AGE   VERSION
ip-172-31-12-177   Ready    <none>                 11h   v1.25.10+k3s-873fc1c7
ip-172-31-14-124   Ready    control-plane,master   12h   v1.25.10+k3s-873fc1c7

Logs from server node:

Jun 15 06:04:45 ip-172-31-14-124 k3s[1790]: Trace[650965794]: ---"Write to database call finished" len:332,err:Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://httpbin.org/status/502?timeout=5s": context deadline exceeded 5001ms (06:04:45.576)
Jun 15 06:04:45 ip-172-31-14-124 k3s[1790]: Trace[650965794]: [5.001292839s] [5.001292839s] END
Jun 15 06:04:45 ip-172-31-14-124 k3s[1790]: I0615 06:04:45.583050    1790 event.go:294] "Event occurred" object="ip-172-31-12-177" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://httpbin.org/status/502?timeout=5s\": context deadline exceeded"