Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.26] - k3s-etcd-snapshot configmap should not contain entries without extra metadata #8859

Closed
brandond opened this issue Nov 14, 2023 · 1 comment
Closed

[Release-1.26] - k3s-etcd-snapshot configmap should not contain entries without extra metadata #8859

brandond opened this issue Nov 14, 2023 · 1 comment
Assignees
@brandond @endawkins

Comments

@brandond
Copy link
Member

Backport fix for k3s-etcd-snapshot configmap should not contain entries without extra metadata
@brandond brandond self-assigned this Nov 14, 2023
@brandond brandond mentioned this issue Nov 16, 2023
Merged
@mdrahman-suse
Copy link

Validated in v1.26.11-rc1+k3s1

Followed through the steps in #8834 (comment)

Replication Results

$ kubectl get nodes,pods -A -o wide
NAME           STATUS   ROLES                       AGE    VERSION         INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/server1   Ready    control-plane,etcd,master   9m1s   v1.26.10+k3s2   172.31.18.182   18.191.83.126   Ubuntu 22.04.2 LTS   5.15.0-1031-aws   containerd://1.7.7-k3s1.26

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-59b4f5bbd5-9lg47                  1/1     Running     0          8m45s   10.42.0.4   server1   <none>           <none>
kube-system   pod/helm-install-traefik-crd-mqsvl            0/1     Completed   0          8m46s   10.42.0.2   server1   <none>           <none>
kube-system   pod/helm-install-traefik-mvjdd                0/1     Completed   2          8m46s   10.42.0.3   server1   <none>           <none>
kube-system   pod/local-path-provisioner-76d776f6f9-jd99f   1/1     Running     0          8m45s   10.42.0.6   server1   <none>           <none>
kube-system   pod/metrics-server-68cf49699b-4t5sc           1/1     Running     0          8m45s   10.42.0.5   server1   <none>           <none>
kube-system   pod/svclb-traefik-e8a0b91f-hx2zn              2/2     Running     0          8m7s    10.42.0.7   server1   <none>           <none>
kube-system   pod/traefik-dd49d7bb6-2xxvs                   1/1     Running     0          8m7s    10.42.0.8   server1   <none>           <none>

$ k3s -v
k3s version v1.26.10+k3s2 (cb5cb555)
go version go1.20.10

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name Location Size Created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267
{"level":"info","ts":"2023-12-01T23:11:06.632056Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267.part"}
{"level":"info","ts":"2023-12-01T23:11:06.634821Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:11:06.63487Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:11:06.6887Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:11:06.702688Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"4.2 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:11:06.702906Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701472267 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267 4177952 2023-12-01T23:11:07Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701472267: '{"name":"on-demand-server1-1701472267","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267","nodeName":"server1","createdAt":"2023-12-01T23:11:07Z","size":4177952,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T23:11:06Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "2711"
  uid: f87128da-f5be-4d39-aad6-3e9c41784e17
kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
configmap/k3s-etcd-snapshot-extra-metadata created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472281
{"level":"info","ts":"2023-12-01T23:11:21.309137Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472281.part"}
{"level":"info","ts":"2023-12-01T23:11:21.312849Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:11:21.312898Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:11:21.368277Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:11:21.547177Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"4.2 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:11:21.54732Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472281"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-server1-1701472281
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701472267 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267 4177952 2023-12-01T23:11:07Z
on-demand-server1-1701472281 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472281 4227104 2023-12-01T23:11:21Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701472267: '{"name":"on-demand-server1-1701472267","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472267","nodeName":"server1","createdAt":"2023-12-01T23:11:07Z","size":4177952,"status":"successful","compressed":false}'
  local-on-demand-server1-1701472281: '{"name":"on-demand-server1-1701472281","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472281","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"server1","createdAt":"2023-12-01T23:11:21Z","size":4227104,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T23:11:06Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "2770"
  uid: f87128da-f5be-4d39-aad6-3e9c41784e17
ubuntu@ip-172-31-18-182:~$

Validation Results

$ kubectl get nodes,pods -A -o wide
NAME           STATUS   ROLES                       AGE     VERSION             INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/server1   Ready    control-plane,etcd,master   4m57s   v1.26.11-rc1+k3s1   172.31.18.182   18.191.83.126   Ubuntu 22.04.2 LTS   5.15.0-1031-aws   containerd://1.7.7-k3s1.26

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-59b4f5bbd5-8zcpz                  1/1     Running     0          4m43s   10.42.0.5   server1   <none>           <none>
kube-system   pod/helm-install-traefik-crd-hpxg8            0/1     Completed   0          4m43s   10.42.0.3   server1   <none>           <none>
kube-system   pod/helm-install-traefik-pclq8                0/1     Completed   1          4m43s   10.42.0.2   server1   <none>           <none>
kube-system   pod/local-path-provisioner-76d776f6f9-d4b82   1/1     Running     0          4m43s   10.42.0.4   server1   <none>           <none>
kube-system   pod/metrics-server-68cf49699b-hq8r4           1/1     Running     0          4m43s   10.42.0.6   server1   <none>           <none>
kube-system   pod/svclb-traefik-0ce2f1fa-lfwkr              2/2     Running     0          4m16s   10.42.0.7   server1   <none>           <none>
kube-system   pod/traefik-dd49d7bb6-n2r8h                   1/1     Running     0          4m16s   10.42.0.8   server1   <none>           <none>

$ k3s -v
k3s version v1.26.11-rc1+k3s1 (d00ad9db)
go version go1.20.11

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name Location Size Created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472850
{"level":"info","ts":"2023-12-01T23:20:49.64016Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472850.part"}
{"level":"info","ts":"2023-12-01T23:20:49.642136Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:20:49.642184Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:20:49.688797Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:20:49.700991Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.5 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:20:49.701083Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472850"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701472850 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472850 3481632 2023-12-01T23:20:50Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
Error from server (NotFound): configmaps "k3s-etcd-snapshots" not found

$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
configmap/k3s-etcd-snapshot-extra-metadata created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472864
{"level":"info","ts":"2023-12-01T23:21:04.276353Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472864.part"}
{"level":"info","ts":"2023-12-01T23:21:04.278822Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T23:21:04.278922Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T23:21:04.319992Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T23:21:04.332193Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.5 MB","took":"now"}
{"level":"info","ts":"2023-12-01T23:21:04.332424Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472864"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-server1-1701472864
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701472850 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472850 3481632 2023-12-01T23:20:50Z
on-demand-server1-1701472864 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472864 3534880 2023-12-01T23:21:04Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701472864: '{"name":"on-demand-server1-1701472864","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701472864","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"server1","createdAt":"2023-12-01T23:21:04Z","size":3534880,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T23:21:04Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "1835"
  uid: 03836bde-0fdc-4b85-af58-b27c03851ab9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@endawkins endawkins

Labels
None yet
Projects
K3s Development
Status: Done Issue
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brandond @mdrahman-suse @endawkins