Plugins system MVP

[CBenoit]

Not yet ready, just opening for interested people and tracking my own progression.

Progress

• Skeleton and API of the PluginsSystem
• Host to content callback test (ready to be implemented, I just need to check wat syntax for passing pointer to function)
• Basic Helix API for registering plugin-defined typable command
• PluginsSystem configuration interface (probably in a [plugins] section in config.toml)

[pickfire]

cc @kirawi

[kirawi]

I haven't inspected the code nor dug into Wasmtime, so these are just my likely very naive thoughts on how to proceed with a plugin system. I believe that helix-plugin should only contain the primitives that would be used by dependents to craft their own respective APIs. An ideal situation would be to cover most of such an API inside helix-view. I have no idea what would be the best way to proceed on that front, however. Hopefully I can offer a better observation tomorrow with some research.

[kirawi]

Upon a closer look, it seems like we're mostly on the same page. I'm not sure about whether we would want witx, however. I would prefer to avoid bindgen-type tools if possible.

Awesome work overall, though!

[TheLostLambda]

This is looking quite cool so far! If you want to avoid code generation, @kirawi, I've got some experience with an alternative method (passing data over JSON) that I chatted about here: #122 (comment)

Happy to help however I can with this! It's exciting work!

[CBenoit]

I haven't inspected the code nor dug into Wasmtime, so these are just my likely very naive thoughts on how to proceed with a plugin system. I believe that helix-plugin should only contain the primitives that would be used by dependents to craft their own respective APIs. An ideal situation would be to cover most of such an API inside helix-view. I have no idea what would be the best way to proceed on that front, however. Hopefully I can offer a better observation tomorrow with some research.

I’ll probably change structure before removing draft and maybe take inspiration from veloren structure a bit.
Now that I think about it helix-plugin sounds like a good name for a Rust crate providing an idiomatic way to write Helix plugins in Rust.

I'm not sure about whether we would want witx, however. I would prefer to avoid bindgen-type tools if possible.

May I ask you why you prefer to avoid witx-bindgen?
Just to clarify, witx is used to describe WASI API in a language-agnostic way.
This is notably used by WASI standard itself to describe the core WASI APIs, then auto-generated and implemented by wasmtime.
This is a sane way to have consistent APIs and implementations across languages / alternatives implementations. I would use cbindgen for the exact same reason if I had to maintain a C API.

Currently it can only generate bindings for Rust on contents (plugins) side, but long term plan is to support more than that.

Alex Crichton in bytecodealliance’s zulip:

On the wasm-content side, witx-bindgen and the general scheme around interface types and the canonical ABI is all being designed explicitly to support multiple languages on all sides, meaning that a wasm module using interface types can be written in any language and the runtime consuming the wasm module can also be any supporting runtime (e.g. wasmtime, a browser, node, ...). Currently I've implemented host support in the form of Rust wasmtime bindings and JS web/node/deno bindings. For the compiled-to-wasm side I've only implemented Rust so far, but I hope to implement at least C in the near future as at least a proof-of-concept. The intention is that writing a language binding generator isn't the hardest thing in the world (but also not a trivial task), so other languages can be done as they come up as well.

(witx-bindgen currently only generating bindings for wasmtime (because WIP) is the reason why plugins system MVP is using wasmtime.)

The other sane approach I can think of, is the one pointed by @TheLostLambda:

If you want to avoid code generation, @kirawi, I've got some experience with an alternative method (passing data over JSON) that I chatted about here: #122 (comment)

But the downside is that it requires serializing and deserializing (copying around) data. I believe the approach is well suited to Zellij, but we can probably do better with witx(-bindgen) in Helix specifically and I think code generation is a good tradeoff here.

Happy to help however I can with this! It's exciting work!

This will be much welcomed! 😄

[e-ivkov]

Is there any ETA for this? Maybe some plans to support plugins in alpha stage?

[e-ivkov]

@CBenoit if you need any help, I can find some time to work on this. I have some experience in integrating WASM for https://github.com/hyperledger/iroha/tree/iroha2-dev

[CBenoit]

@CBenoit if you need any help, I can find some time to work on this. I have some experience in integrating WASM for https://github.com/hyperledger/iroha/tree/iroha2-dev

This is greatly welcomed. No real work happened recently and a lot remains to be done. I’m somewhat keeping an eye on the original issue, but lack the time to actually work on it 😕

[gavynriebau]

I've also started work on a wasm plugin system for helix and have raised a draft PR at #2949.

I started this second effort before I discovered this already open PR. I started it because it sounded like an interesting challenge and my branch is at a point where I wanted to share what I've done in the hope that it will benefit the existing efforts here.

I'd be keen to exchange ideas or answer any questions about what I've done so far which can be summarized as:

• Files located under ~/.config/helix/plugins and having .wasm extension are automatically loaded on helix start
• Plugins are executed within a wasmer runtime
• Messages are defined in a protobuf file and rust types are automatically generated for these messages
• Whenever a key is pressed a KeyEvent message is sent to the plugin
• Messages are sent to plugins by converting the protobuf type to raw bytes, allocating plugin memory, copying the bytes to plugin memory, calling an exported handle_event function with a pointer to the bytes as well as the size
• An example rust plugin is located here

[jgarvin]

Since WASM is compiled, is there a reason plugins couldn't define functions that are just called directly, without going through any serialization? I don't know how much it matters for a text editor, but protobuf stop bit based encoding/decoding is very slow compared to just regular structs. Also possibly more relevant is that it's much more complex to debug (reading a hexdump of a struct is much easier). The rykv crate is very polished and gives zero copy structs.

[kirawi]

The issue is that Wasm doesn't have a stable ABI for other languages to interface with, and rkyv makes it a non-goal to implement serializers for other languages because it depends upon a type system like Rust or C++20 for it to work. I personally
don't think that using protobuf is a good way forward.

[arendjr]

@kirawi Would you be open to using something like fp-bindgen instead? https://github.com/fiberplane/fp-bindgen

It’s still a WASM bindgen tool, but it uses Rust types as the source of truth instead of a specialized DSL. This means that any changes to those types don’t force you to regenerate anything unless you’re specifically working on the plugin bindings, i.e. it can help quite a bit on the workflow side of things.