[release-1.24] Failing secret webhook causes unexpected behavior

[brandond]

• Failing secret webhook causes unexpected behavior #7654 backport

[endawkins]

Validated on branch 1.24 with commit 20e2461 / version 1.24

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-30-17 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24 01:58:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Cluster Configuration:

1 server 1 agent

Config.yaml:

server:

cluster-init: true
write-kubeconfig-mode: 644

agent:

server: "https://<serverIP>:6443"
token: test

Additional files

web hook_bad.yaml:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: rancher.cattle.io
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    url: https://httpbin.org/status/502
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rancher.cattle.io.secrets
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - secrets
    scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 5

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s on the server
3. Run

k apply -f web hook_bad.yaml

4. Join the agent node

Replication Results:

• k3s version used for replication:

k3s version v1.24.14+k3s1 (313a3b99)
go version go1.19.9

server node logs:

Jun 22 07:48:23 ip-172-31-30-17 k3s[35174]: time="2023-06-22T07:48:23Z" level=error msg="Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: an error on the server (\"\") has prevented the request from succeeding"


agent node logs:

Jun 22 07:49:17 ip-172-31-17-218 k3s[6095]: time="2023-06-22T07:49:17Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"

Validation Results:

• k3s version used for validation:

k3s version v1.24.15-rc1+k3s1 (20e24611)
go version go1.19.10

ubuntu@ip-172-31-30-17:~$ kubectl get nodes,pods -A -o wide
NAME                    STATUS   ROLES                       AGE     VERSION             INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/ip-172-31-17-218   Ready    <none>                      69s     v1.24.15-rc1+k3s1   172.31.17.218   <none>        Ubuntu 22.04.2 LTS   5.19.0-1025-aws   containerd://1.7.1-k3s1
node/ip-172-31-30-17    Ready    control-plane,etcd,master   4m23s   v1.24.15-rc1+k3s1   172.31.30.17    <none>        Ubuntu 22.04.2 LTS   5.19.0-1025-aws   containerd://1.7.1-k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE     IP          NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-74448699cf-d24dc                  1/1     Running     0          4m8s    10.42.0.6   ip-172-31-30-17    <none>           <none>
kube-system   pod/helm-install-traefik-crd-s8w99            0/1     Completed   0          4m9s    10.42.0.3   ip-172-31-30-17    <none>           <none>
kube-system   pod/helm-install-traefik-rwdh5                0/1     Completed   1          4m9s    10.42.0.2   ip-172-31-30-17    <none>           <none>
kube-system   pod/local-path-provisioner-597bc7dccd-5txqn   1/1     Running     0          4m8s    10.42.0.5   ip-172-31-30-17    <none>           <none>
kube-system   pod/metrics-server-749b574fc-p69sv            1/1     Running     0          4m8s    10.42.0.4   ip-172-31-30-17    <none>           <none>
kube-system   pod/svclb-traefik-a68d5d33-7k6sx              2/2     Running     0          68s     10.42.1.2   ip-172-31-17-218   <none>           <none>
kube-system   pod/svclb-traefik-a68d5d33-prx4z              2/2     Running     0          3m52s   10.42.0.7   ip-172-31-30-17    <none>           <none>
kube-system   pod/traefik-7467b667d9-knqgr                  1/1     Running     0          3m53s   10.42.0.8   ip-172-31-30-17    <none>           <none>

Additional context / logs:

N/A