K3s uses incorrect wasm shims names #9650

brandond · 2024-03-05T21:13:43Z

K3s should use correct binary names and the associated RuntimeType. There was quite a mixup between v1 and v2 shims.

VestigeJ · 2024-03-06T18:35:18Z

Interesting I didn't hit this initially because I was writing out the runtime information on the nodes manually.

VestigeJ · 2024-03-06T19:05:30Z

##Environment Details
Reproduced using VERSION=v1.29.2+k3s1
Validated using COMMIT=6f331ea7b520095785a3a4ca30f8c75c69a12105

Infrastructure

Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE   VERSION
ip-1-1-1-158       Ready    control-plane,etcd,master   74s   v1.29.2+k3s-6f331ea7

Config.yaml:

node-external-ip: 1.1.1.158
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
profile: cis
protect-kernel-defaults: true
cluster-init: true

Reproduction

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ VERSION=v1.29.2+k3s1
$ vim deploy.yaml
$ vim spin-deploy.yaml
$ vim middleware.yaml
$ wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.11.1/containerd-wasm-shims-v2-spin-linux-x86_64.tar.gz
$ wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.11.1/containerd-wasm-shims-v1-slight-linux-x86_64.tar.gz
$ //tar and cp the binaries to your PATH location
$ sudo INSTALL_K3S_VERSION=$VERSION INSTALL_K3S_EXEC=server ./install-k3s.sh
$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
$ ls /usr/local/bin/
$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
$ COMMIT=6f331ea7b520095785a3a4ca30f8c75c69a12105
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=server ./install-k3s.sh
$ ls /usr/local/bin/
$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
$ kubectl get pods
$

Results:

Existing release results - all binaries installed before installing k3s

$ ls /usr/local/bin/

containerd-shim-lunatic-v1  containerd-shim-slight-v1  containerd-shim-spin-v2  containerd-shim-wws-v1  crictl  ctr  k3s  k3s-killall.sh  k3s-uninstall.sh  kubectl

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true



[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/a3b46c0299091b71bfcc617b1e1fec1845c13bdd848584ceb39d2e700e702a4b/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"






[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic"]
  runtime_type = "io.containerd.lunatic.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic".options]
  BinaryName = "/usr/local/bin/containerd-shim-lunatic-v1"
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."slight"]
  runtime_type = "io.containerd.slight.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."slight".options]
  BinaryName = "/usr/local/bin/containerd-shim-slight-v1"
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."wws"]
  runtime_type = "io.containerd.wws.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."wws".options]
  BinaryName = "/usr/local/bin/containerd-shim-wws-v1"
  SystemdCgroup = true

After latest COMMIT_ID install

$ ls /usr/local/bin/

containerd-shim-lunatic-v1  containerd-shim-slight-v1  containerd-shim-spin-v2  containerd-shim-wws-v1  crictl  ctr  k3s  k3s-killall.sh  k3s-uninstall.sh  kubectl

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true



[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/0bc089d320542b56f300f222c70a28733d12f7d148623d2f8e4e2c915ca8cf29/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"






[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic"]
  runtime_type = "io.containerd.lunatic.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic".options]
  BinaryName = "/usr/local/bin/containerd-shim-lunatic-v1"
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."slight"]
  runtime_type = "io.containerd.slight.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."slight".options]
  BinaryName = "/usr/local/bin/containerd-shim-slight-v1"
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/usr/local/bin/containerd-shim-spin-v2"
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."wws"]
  runtime_type = "io.containerd.wws.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."wws".options]
  BinaryName = "/usr/local/bin/containerd-shim-wws-v1"
  SystemdCgroup = true

$ kgp

NAME                           READY   STATUS    RESTARTS   AGE
wasm-slight-6bb5c5bf49-ztlz9   1/1     Running   0          8s
wasm-spin-7cb47dbd9b-wss8b     1/1     Running   0          8s

$ export PUBLIC_IP=$(kubectl get ingress wasm-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

$ curl -v http://$PUBLIC_IP/slight/hello

*   Trying 1.1.1.158:80...
* Connected to 1.1.1.158 (1.1.1.158) port 80 (#0)
> GET /slight/hello HTTP/1.1
> Host: 1.1.1.158
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept: */*
< Accept-Encoding: gzip
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Methods: *
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: *
< Content-Length: 12
< Date: Wed, 06 Mar 2024 18:58:55 GMT
< Host: 1.1.1.158
< User-Agent: curl/8.0.1
< X-Forwarded-For: 10.42.0.1
< X-Forwarded-Host: 1.1.1.158
< X-Forwarded-Port: 80
< X-Forwarded-Prefix: /slight
< X-Forwarded-Proto: http
< X-Forwarded-Server: traefik-f4564c4f4-xnqz7
< X-Real-Ip: 10.42.0.1
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 1.1.1.158 left intact
hello world!

$ curl -v http://$PUBLIC_IP/spin/go-hello

*   Trying 1.1.1.158:80...
* Connected to 1.1.1.158 (1.1.1.158) port 80 (#0)
> GET /spin/go-hello HTTP/1.1
> Host: 1.1.1.158
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 723
< Content-Type: text/plain
< Date: Wed, 06 Mar 2024 18:59:02 GMT
< Foo: bar
<
== REQUEST ==
URL:     http://1.1.1.158/go-hello
Method:  GET
Headers:
  "Accept": "*/*"
  "X-Forwarded-Host": "1.1.1.158"
  "X-Forwarded-Proto": "http"
  "X-Forwarded-Server": "traefik-f4564c4f4-xnqz7"
  "Spin-Matched-Route": "/go-hello"
  "Spin-Component-Route": "/go-hello"
  "Host": "1.1.1.158"
  "X-Forwarded-Prefix": "/spin"
  "Spin-Path-Info": ""
  "Spin-Base-Path": "/"
  "Spin-Raw-Component-Route": "/go-hello"
  "Spin-Client-Addr": "10.42.0.8:49262"
  "User-Agent": "curl/8.0.1"
  "Accept-Encoding": "gzip"
  "Spin-Full-Url": "http://1.1.1.158/go-hello"
  "X-Forwarded-For": "10.42.0.1"
  "X-Forwarded-Port": "80"
  "X-Real-Ip": "10.42.0.1"
Body:
== RESPONSE ==
Hello Spin Shim!
* Connection #0 to host 1.1.1.158 left intact

$ curl -v http://$PUBLIC_IP/spin/hello

*   Trying 1.1.1.158:80...
* Connected to 1.1.1.158 (1.1.1.158) port 80 (#0)
> GET /spin/hello HTTP/1.1
> Host: 1.1.1.158
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 06 Mar 2024 18:59:06 GMT
< Transfer-Encoding: chunked
<
* Connection #0 to host 1.1.1.158 left intact

brandond self-assigned this Mar 5, 2024

brandond added this to the v1.29.3+k3s1 milestone Mar 5, 2024

This was referenced Mar 5, 2024

fix: use correct wasm shims names #9519

Merged

[Release-1.28] - K3s uses incorrect wasm shims names #9651

Closed

[Release-1.27] - K3s uses incorrect wasm shims names #9652

Closed

endawkins assigned VestigeJ Mar 6, 2024

brandond mentioned this issue Mar 7, 2024

[Release-1.26] - K3s uses incorrect wasm shims names #9690

Closed

VestigeJ closed this as completed Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

K3s uses incorrect wasm shims names #9650

K3s uses incorrect wasm shims names #9650

brandond commented Mar 5, 2024

VestigeJ commented Mar 6, 2024

VestigeJ commented Mar 6, 2024

Reproduction

K3s uses incorrect wasm shims names #9650

K3s uses incorrect wasm shims names #9650

Comments

brandond commented Mar 5, 2024

VestigeJ commented Mar 6, 2024

VestigeJ commented Mar 6, 2024

Reproduction