Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k3s-etcd-snapshot configmap should not contain entries without extra metadata #8834

Closed
brandond opened this issue Nov 14, 2023 · 1 comment
Closed

k3s-etcd-snapshot configmap should not contain entries without extra metadata #8834

brandond opened this issue Nov 14, 2023 · 1 comment
Assignees
@brandond @mdrahman-suse @endawkins
Milestone
v1.28.4+k3s1

Comments

@brandond
Copy link
Member

K3s tracking issue for

The snapshot controller shouldn't create configmap entries for snapshots that have empty extra metadata; this confuses Rancher.
@brandond brandond self-assigned this Nov 14, 2023
@brandond brandond added this to the v1.27.8+k3s1 milestone Nov 14, 2023
@brandond brandond mentioned this issue Nov 14, 2023
Merged
@brandond brandond modified the milestones: v1.27.8+k3s1, v1.28.4+k3s1 Nov 14, 2023
This was referenced Nov 14, 2023
Closed
Closed
Closed
@mdrahman-suse mdrahman-suse self-assigned this Dec 1, 2023
@mdrahman-suse
Copy link

mdrahman-suse commented Dec 1, 2023 •
edited
Loading

Validated in v1.28.4-rc1+k3s1

Environment

Ubuntu 22.04, 1 server
  • Config.yaml
write-kubeconfig-mode: 644
cluster-init: true
token: summer
node-name: server1

Testing Steps

  • Copy Config.yaml in /etc/rancher/k3s/config.yaml and Install k3s
  • Run the commands below:
k3s etcd-snapshot save
k3s etcd-snapshot list
  • Confirm snapshot is listed
kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
  • Confirm empty or does not exist
kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
k3s etcd-snapshot save
k3s etcd-snapshot list
  • Confirm both snapshots are listed
kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
  • Confirm only second (or latest) snapshot is listed, not all

Replication Result

$ kubectl get nodes,pods -A -o wide
NAME           STATUS   ROLES                       AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/server1   Ready    control-plane,etcd,master   3m44s   v1.28.3+k3s2   172.31.18.182   <none>        Ubuntu 22.04.2 LTS   5.15.0-1031-aws   containerd://1.7.7-k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-6799fbcd5-sb865                   1/1     Running     0          3m29s   10.42.0.6   server1   <none>           <none>
kube-system   pod/helm-install-traefik-bdnnt                0/1     Completed   1          3m29s   10.42.0.5   server1   <none>           <none>
kube-system   pod/helm-install-traefik-crd-f4njh            0/1     Completed   0          3m29s   10.42.0.2   server1   <none>           <none>
kube-system   pod/local-path-provisioner-84db5d44d9-v6pz5   1/1     Running     0          3m29s   10.42.0.4   server1   <none>           <none>
kube-system   pod/metrics-server-67c658944b-nrjpc           1/1     Running     0          3m29s   10.42.0.3   server1   <none>           <none>
kube-system   pod/svclb-traefik-2e884bcf-wls7b              2/2     Running     0          2m44s   10.42.0.8   server1   <none>           <none>
kube-system   pod/traefik-f4564c4f4-zqtw9                   1/1     Running     0          2m44s   10.42.0.7   server1   <none>           <none>

$ k3s -v
k3s version v1.28.3+k3s2 (bbafb86e)
go version go1.20.10

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
Name Location Size Created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447
{"level":"info","ts":"2023-12-01T21:34:06.949343Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447.part"}
{"level":"info","ts":"2023-12-01T21:34:06.952698Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T21:34:06.953123Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T21:34:06.994839Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T21:34:07.094617Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.3 MB","took":"now"}
{"level":"info","ts":"2023-12-01T21:34:07.094711Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701466447 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447 3297312 2023-12-01T21:34:07Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701466447: '{"name":"on-demand-server1-1701466447","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447","nodeName":"server1","createdAt":"2023-12-01T21:34:07Z","size":3297312,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T21:34:07Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "1469"
  uid: feb34dcf-f570-4212-909b-ee4a375e2320

$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
configmap/k3s-etcd-snapshot-extra-metadata created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466500
{"level":"info","ts":"2023-12-01T21:34:59.576192Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466500.part"}
{"level":"info","ts":"2023-12-01T21:34:59.580915Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T21:34:59.580977Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T21:34:59.622174Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T21:34:59.805673Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.4 MB","took":"now"}
{"level":"info","ts":"2023-12-01T21:34:59.805762Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466500"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-server1-1701466500
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701466447 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447 3297312 2023-12-01T21:34:07Z
on-demand-server1-1701466500 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466500 3416096 2023-12-01T21:35:00Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701466447: '{"name":"on-demand-server1-1701466447","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466447","nodeName":"server1","createdAt":"2023-12-01T21:34:07Z","size":3297312,"status":"successful","compressed":false}'
  local-on-demand-server1-1701466500: '{"name":"on-demand-server1-1701466500","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701466500","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"server1","createdAt":"2023-12-01T21:35:00Z","size":3416096,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T21:34:07Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "1621"
  uid: feb34dcf-f570-4212-909b-ee4a375e2320

Validation Results

$ kubectl get nodes,pods -A -o wide
NAME           STATUS   ROLES                       AGE     VERSION            INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
node/server1   Ready    control-plane,etcd,master   3m27s   v1.28.4-rc1+k3s1   172.31.18.182   <none>        Ubuntu 22.04.2 LTS   5.15.0-1031-aws   containerd://1.7.7-k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-6799fbcd5-pdsqk                   1/1     Running     0          3m12s   10.42.0.4   server1   <none>           <none>
kube-system   pod/helm-install-traefik-4gn6l                0/1     Completed   1          3m13s   10.42.0.3   server1   <none>           <none>
kube-system   pod/helm-install-traefik-crd-2kl7t            0/1     Completed   0          3m13s   10.42.0.2   server1   <none>           <none>
kube-system   pod/local-path-provisioner-84db5d44d9-p68rm   1/1     Running     0          3m12s   10.42.0.6   server1   <none>           <none>
kube-system   pod/metrics-server-67c658944b-2dn6m           1/1     Running     0          3m12s   10.42.0.5   server1   <none>           <none>
kube-system   pod/svclb-traefik-7691f47f-q84nl              2/2     Running     0          2m52s   10.42.0.7   server1   <none>           <none>
kube-system   pod/traefik-f4564c4f4-hs5lk                   1/1     Running     0          2m52s   10.42.0.8   server1   <none>           <none>

$ k3s -v
k3s version v1.28.4-rc1+k3s1 (3f237230)
go version go1.20.11

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
Name Location Size Created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467256
{"level":"info","ts":"2023-12-01T21:47:36.153111Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467256.part"}
{"level":"info","ts":"2023-12-01T21:47:36.157183Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T21:47:36.157229Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T21:47:36.196859Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T21:47:36.296218Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.2 MB","took":"now"}
{"level":"info","ts":"2023-12-01T21:47:36.296312Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467256"}
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701467256 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467256 3153952 2023-12-01T21:47:36Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
Error from server (NotFound): configmaps "k3s-etcd-snapshots" not found

$ kubectl create configmap -n kube-system k3s-etcd-snapshot-extra-metadata --from-literal=foo=bar
configmap/k3s-etcd-snapshot-extra-metadata created

$ sudo k3s etcd-snapshot save
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467322
{"level":"info","ts":"2023-12-01T21:48:41.723022Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467322.part"}
{"level":"info","ts":"2023-12-01T21:48:41.726253Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-12-01T21:48:41.726504Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-12-01T21:48:41.77451Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-12-01T21:48:41.857923Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.3 MB","took":"now"}
{"level":"info","ts":"2023-12-01T21:48:41.85802Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467322"}
INFO[0000] Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-server1-1701467322
INFO[0000] Reconciling ETCDSnapshotFile resources
INFO[0000] Reconciliation of ETCDSnapshotFile resources complete

$ sudo k3s etcd-snapshot list
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --cluster-init found in config.yaml, skipping
WARN[0000] Unknown flag --token found in config.yaml, skipping
Name                         Location                                                                     Size    Created
on-demand-server1-1701467256 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467256 3153952 2023-12-01T21:47:36Z
on-demand-server1-1701467322 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467322 3276832 2023-12-01T21:48:42Z

$ kubectl get configmap -n kube-system k3s-etcd-snapshots -o yaml
apiVersion: v1
data:
  local-on-demand-server1-1701467322: '{"name":"on-demand-server1-1701467322","location":"file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server1-1701467322","metadata":"eyJmb28iOiJiYXIifQ==","nodeName":"server1","createdAt":"2023-12-01T21:48:42Z","size":3276832,"status":"successful","compressed":false}'
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-01T21:48:41Z"
  name: k3s-etcd-snapshots
  namespace: kube-system
  resourceVersion: "1468"
  uid: 2908469f-1e59-4ac3-a05a-1e602d36a51c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@mdrahman-suse mdrahman-suse

@endawkins endawkins

Labels
None yet
Projects
K3s Development
Status: Done Issue
Milestone
v1.28.4+k3s1
Development

No branches or pull requests
3 participants
@brandond @mdrahman-suse @endawkins