Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.27] - k3s-etcd-snapshot configmap should not contain entries without extra metadata #8858

Closed
brandond opened this issue Nov 14, 2023 · 1 comment
Closed

[Release-1.27] - k3s-etcd-snapshot configmap should not contain entries without extra metadata #8858

brandond opened this issue Nov 14, 2023 · 1 comment
Assignees
@brandond @mdrahman-suse @endawkins

Comments

@brandond
Copy link
Member

Backport fix for k3s-etcd-snapshot configmap should not contain entries without extra metadata
@brandond brandond self-assigned this Nov 14, 2023
@brandond brandond mentioned this issue Nov 16, 2023
Merged
@mdrahman-suse
Copy link

Validated in v1.27.8-rc1+k3s1

Followed through the steps mentioned in #8834 (comment)

Replication Results

$ kubectl get nodes,pods -A -o wide
NAME           STATUS   ROLES                       AGE     VERSION        INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/server1   Ready    control-plane,etcd,master   6m30s   v1.27.7+k3s2   172.31.18.182   18.191.83.126   Ubuntu 22.04.2 LTS   5.15.0-1031-aws   containerd://1.7.7-k3s1.27

NAMESPACE     NAME                                         READY   STATUS      RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-77ccd57875-p7rcp                 1/1     Running     0          6m15s   10.42.0.5   server1   <none>           <none>
kube-system   pod/helm-install-traefik-crd-bpms5           0/1     Completed   0          6m16s   10.42.0.3   server1   <none>           <none>
kube-system   pod/helm-install-traefik-zck6h               0/1     Completed   1          6m16s   10.42.0.6   server1   <none>           <none>
kube-system   pod/local-path-provisioner-957fdf8bc-r9x5t   1/1     Running     0          6m15s   10.42.0.4   server1   <none>           <none>
kube-system   pod/metrics-server-648b5df564-lj5p7          1/1     Running     0          6m15s   10.42.0.2   server1   <none>           <none>
kube-system   pod/svclb-traefik-34da65fa-d4h74             2/2     Running     0          5m43s   10.42.0.7   server1   <none>           <none>
kube-system   pod/traefik-768bdcdcdd-g5vqx                 1/1     Running     0          5m43s   10.42.0.8   server1   <none>           <none>

$ k3s -v
k3s version v1.27.7+k3s2 (575bce76)
go version go1.20.10

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name Location Size Created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644
{"level":"info","ts":"2023-12-01T22:27:24.390359Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644.part"}
{"level":"info","ts":"2023-12-01T22:27:24.39714Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T22:27:24.397234Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T22:27:24.449937Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T22:27:24.551415Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.8 MB","took":"now"}
{"level":"info","ts":"2023-12-01T22:27:24.551508Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701469644 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644 3776544 2023-12-01T22:27:24Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701469644: '{"name":"on-demand-server1-1701469644","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644","nodeName":"server1","createdAt":"2023-12-01T22:27:24Z","size":3776544,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T22:27:24Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "2187"
  uid: e14a7d8b-1687-4bbd-935b-cfebd33b4261

$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
configmap/k3s-etcd-snapshot-extra-metadata created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469685
{"level":"info","ts":"2023-12-01T22:28:05.113563Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469685.part"}
{"level":"info","ts":"2023-12-01T22:28:05.119565Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T22:28:05.119808Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T22:28:05.171602Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T22:28:05.290806Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2023-12-01T22:28:05.290943Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469685"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-server1-1701469685
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701469644 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644 3776544 2023-12-01T22:27:24Z
on-demand-server1-1701469685 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469685 3899424 2023-12-01T22:28:05Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701469644: '{"name":"on-demand-server1-1701469644","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469644","nodeName":"server1","createdAt":"2023-12-01T22:27:24Z","size":3776544,"status":"successful","compressed":false}'
  local-on-demand-server1-1701469685: '{"name":"on-demand-server1-1701469685","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701469685","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"server1","createdAt":"2023-12-01T22:28:05Z","size":3899424,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T22:27:24Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "2346"
  uid: e14a7d8b-1687-4bbd-935b-cfebd33b4261

Validation Results

$ kubectl get nodes,pods -A -o wide
NAME           STATUS   ROLES                       AGE     VERSION            INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/server1   Ready    control-plane,etcd,master   7m31s   v1.27.8-rc1+k3s1   172.31.18.182   18.191.83.126   Ubuntu 22.04.2 LTS   5.15.0-1031-aws   containerd://1.7.7-k3s1.27

NAMESPACE     NAME                                         READY   STATUS      RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-77ccd57875-gb8fn                 1/1     Running     0          7m16s   10.42.0.2   server1   <none>           <none>
kube-system   pod/helm-install-traefik-8bvzb               0/1     Completed   1          7m16s   10.42.0.6   server1   <none>           <none>
kube-system   pod/helm-install-traefik-crd-t2wq6           0/1     Completed   0          7m16s   10.42.0.3   server1   <none>           <none>
kube-system   pod/local-path-provisioner-957fdf8bc-7szdr   1/1     Running     0          7m16s   10.42.0.5   server1   <none>           <none>
kube-system   pod/metrics-server-648b5df564-fskht          1/1     Running     0          7m16s   10.42.0.4   server1   <none>           <none>
kube-system   pod/svclb-traefik-d10feaf5-qm4q6             2/2     Running     0          6m45s   10.42.0.7   server1   <none>           <none>
kube-system   pod/traefik-768bdcdcdd-djjqr                 1/1     Running     0          6m45s   10.42.0.8   server1   <none>           <none>

$ k3s -v
k3s version v1.27.8-rc1+k3s1 (0abb8a5a)
go version go1.20.11

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name Location Size Created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470508
{"level":"info","ts":"2023-12-01T22:41:47.903954Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470508.part"}
{"level":"info","ts":"2023-12-01T22:41:47.906176Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T22:41:47.90627Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T22:41:47.957993Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T22:41:47.97087Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2023-12-01T22:41:47.970971Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470508"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701470508 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470508 3948576 2023-12-01T22:41:48Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
Error from server (NotFound): configmaps "k3s-etcd-snapshots" not found

$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
configmap/k3s-etcd-snapshot-extra-metadata created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470509
{"level":"info","ts":"2023-12-01T22:41:48.766511Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470509.part"}
{"level":"info","ts":"2023-12-01T22:41:48.768964Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T22:41:48.769014Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T22:41:48.81898Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T22:41:48.830853Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"4.0 MB","took":"now"}
{"level":"info","ts":"2023-12-01T22:41:48.831196Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470509"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-server1-1701470509
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701470508 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470508 3948576 2023-12-01T22:41:48Z
on-demand-server1-1701470509 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470509 3969056 2023-12-01T22:41:49Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701470509: '{"name":"on-demand-server1-1701470509","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701470509","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"server1","createdAt":"2023-12-01T22:41:49Z","size":3969056,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T22:41:48Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "2350"
  uid: 860eb769-782c-44e7-ada6-2936dfb6831b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@mdrahman-suse mdrahman-suse

@endawkins endawkins

Labels
None yet
Projects
K3s Development
Status: Done Issue
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brandond @mdrahman-suse @endawkins