[hack week] Add containerd runwasi wasmtime shims

[brandond]

Proposed Changes

• Refactor the build scripts a bit
• Add containerd runwasi wasmtime shims
• Add RuntimeClass manifests for all supported runtimes (including the autodetected nvidia runtimes)

Note that this currently balloons the k3s build time by several minutes, due to the extra time spent building the rust crates.

Types of Changes

new feature

Verification

Run a WebAssembly image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-wasi
  labels:
    app: example-wasi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-wasi
  template:
    metadata:
      labels:
        app: example-wasi
      annotations:
        module.wasm.image/variant: compat-smart
    spec:
      runtimeClassName: wasmtime
      containers:
      - name: example-wasi
        image: index.docker.io/wasmedge/example-wasi:latest
        imagePullPolicy: IfNotPresent

kubectl logs -l app=example-wasi --tail=-1

Testing

Linked Issues

User-Facing Change

K3s now supports execution of  wasm images via the wasmtime engine.

Further Comments

Caveats:

• runwasi also supports wasmedge, but that requires llvm and lld (the llvm linker) which isn't available on s390x, so I have skipped it for now.
• Only the wasmtime runtime works, wasmtimed (single runtime process shared by all the pods) appears to be broken:
  E0201 23:38:33.301847 53 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err=<rpc error: code = Unknown desc = failed to create containerd task: failed to start shim: start failed: containerd-shim-wasmtimed-v1: Ttrpc(RpcStatus(code: NOT_FOUND message: "/runwasi.services.sandbox.v1.Manager/Connect is not supported")): exit status 1: unknown > pod="default/example-wasi-6b754d67db-htqwc"
  I'm poking upstream about that.

[VestigeJ]

Do you want to pull the wasmtimed runtimeclass until that's working better or leave it in?

[brandond]

oh I'm not planning on merging any of this at the moment, I'm going to keep it around as-is while upstream works on the shims.

[VestigeJ]

In that case I may try to add "runu"

[juntao]

Hi @brandond

Very excited to see k3s supporting Wasm containers!

I am with the WasmEdge project. Just to clarify that WasmEdge has no runtime dependency on LLVM. In fact, one could build WasmEdge using GCC, which is available on s360x. We are using LLVM for AOT compilation. But the "Wasm image" typically only contains the Wasm bytecode (for interpreter) or the AOT compiled native code (done on another machine). In either case, the deployment machine does not need LLVM.

One of the unique features of WasmEdge is that it enables developers to build complete microservices -- HTTP servers, web service clients, pooled database connections, messaging queue clients etc. All using libraries they are already familiar with (eg tokio / hyper on Rust, node / fetch() on JavaScript). See examples here:

https://github.com/docker/awesome-compose/tree/master/wasmedge-mysql-nginx

https://github.com/docker/awesome-compose/tree/master/wasmedge-kafka-mysql

WasmEdge provides first-class support for AI inference frameworks, such as Tensorflow, PyTorch, and OpenVINO.

https://github.com/second-state/WasmEdge-WASINN-examples

Finally, WasmEdge can run JavaScript apps. It supports many node.js APIs and even AI inference APIs.

https://wasmedge.org/book/en/write_wasm/js.html

I could go on ... We would love to see WasmEdge supported in k3s. Thank you!

Michael Yuan - maintainer of WasmEdge

[brandond]

@juntao thanks for stopping by! Do you have any tips on building without a working llvm/lld for the target arch? We aim to ship static binaries for arm/arm64/amd64/s390x and I was having a very hard time finding a distro with working static libs for lld on s390x. My preference would be to build in either buildroot or alpine, given that's where we're doing our existing work.

At the moment we're also using musl instead of glibc, but it appears that some of the rust ecosystem may not play nicely with that either.

Are there any downsides to omitting llvm from the wasmedge build? Does it just break .wit support, requiring the image to contain only compiled to bytecode?

[hydai]

Do you have any tips on building without a working llvm/lld for the target arch?

Hi, the only thing you need is to set WASMEDGE_BUILD_AOT_RUNTIME=OFF in the cmake stage. This will disable the LLVM/LLD dependencies.

[brandond]

@hydai thanks! I was looking at https://wasmedge.org/book/en/contribute/build_from_src.html and https://wasmedge.org/book/en/contribute/build_from_src/linux.html but it wasn't clear from there that turning off WASMEDGE_BUILD_AOT_RUNTIME eliminates the LLVM dependencies, but that makes sense.

It looks like I'll need to use a gnu toolchain for rust on arm and s390x, so I'lll probably have to move the runwasi build out to another pipeline... which is fine, as they were probably too time consuming to do in the main K3s build anyway.

[hydai]

It looks like I'll need to use a gnu toolchain for rust on arm and s390x, so I'lll probably have to move the runwasi build out to another pipeline... which is fine, as they were probably too time consuming to do in the main K3s build anyway.

I am not familiar with the s390x arch. Please let me know if you encounter any issues :-)
We(WasmEdge) would like to support more arches if possible.

[brandond]

@hydai I have given up on s390x due to lack of rust support for musl libc on that platform, but I'm running into problems building for 32bit arm which I would expect to work. I opened WasmEdge/WasmEdge#2254 if you wouldn't mind taking a look.

[hydai]

Hi @brandond
Thanks for the report. I am going to dig into this issue.
I have a question: Does k3s use musl libc only? If so, I should also need to check the compatibility between our current implementation (based on glibc) and the musl libc.

[brandond]

@hydai yes, we use musl for everything. Our "root" (userspace binaries that ship with k3s) are derived from buildroot, while the golang bits are built on alpine. Everything is statically linked.

[devigned]

It sounds like it would be helpful if runwasi provided prebuilt bins with some variety of configurations. What would help most? We'd be happy to help make this process easier.

Also have you consider the spin and slight shims via https://github.com/deislabs/containerd-wasm-shims? We are including these in AKS and in Cluster API (CAPZ documentation is forthcoming, but the bits are in image-builder via kubernetes-sigs/image-builder#1037).