Add an embeddable execution mode

[nex3]

It's desirable to be able to use Dart Sass from various different language environments, with functions and importers defined in the host language, while still getting the language semantics of Dart Sass and the performance of the Dart VM. We could theoretically expose the VM plus Dart Sass as a C API, but figuring out how to get that to build reliably (let alone efficiently) seems like a nightmare.

A much easier solution would be to create an executable that uses stdin/stdout to allow a caller to define functions and importers. This would require defining a protocol of some sort—probably either JSON- or protobuf-based—for communication and for serializing Sass values.

This should probably be in a separate package from Dart Sass proper, but that package doesn't exist yet so I'm filing this here for now.

[xzyfer]

We've also considered the latter in Node Sass as an alternative to doing native binding. Would be interested to see where this discussion goes, and if it can be general enough for other language implementations.

[nex3]

We should definitely coordinate to come up with a good, cross-implementation protocol.

[nex3]

Upon consideration, I don't expect that I'll personally have time to create a Ruby or JS wrapper using this. It's still something I think would be valuable, but I want to wait to start in on it in earnest until we have someone signed up to write at least one wrapper that makes use of it.

[nex3]

@strychnide has volunteered to create/maintain the Node side of this, so we can start thinking about it more seriously. I think the first question is, what technology should we use? The two main criteria as I see them are:

• Portability. We want to make it straightforward for any language to embed, which probably means using well-defined protocols that have existing libraries in many languages. We also want to make sure it's possible for LibSass to eventually expose the same API to aid its own embedability.

• Efficiency. The whole point of embedding the Dart VM rather than using the JS compiled version is for its increased speed versus V8. We should aim to lose as little of this speed as possible in overhead, even for stylesheets that make many native function or importer calls.

I think protocol buffers as the transport method is a good choice for both of these. Protobufs are much more compact and quicker to de/serialize than JSON (the other obvious option), and they have client libraries for Dart, Node.js, and Ruby.

At a higher level, we also need a protocol for defining the calls that the embedder can make into the embedded executable. This is trickier than most RPC systems, because we want RPCs to flow both ways. We need the embedder to be able to ask the executable to compile files, and we need the executable to be able to ask the embedder to run functions and importers. I think we have two options here:

1. Use gRPC. The big benefit here is that this is totally standard, and again has existing libraries for Dart, Node.js, and Ruby. However, it makes the strong assumption that RPCs are made in one direction to a remote server. It requires HTTP/2, which I suspect (but have not verified) is slower than communicating directly over stdin/stdout; and it doesn't have explicit support for callbacks. We could fake them with bidirectional streaming, but at that point we'd be halfway to rolling our own protocol, since we couldn't model and server-to-client requests as proper RPCs.

2. Roll our own. This would require more specification work on our part as well as more work for users to generate, but it would let us communicate directly over stdin/stdout and we could easily support callbacks. We could also potentially base this on existing specs that don't use protobufs; for example, JSON-RPC 2 supports bidirectional calls over a generic byte stream.

My intuition is that option 2 is best, but I'm open to disagreement (or another alternative I haven't considered).

[ghost]

Using gRPC sounds like a massive overshoot to me, HTTP/2 is not meant for this kind of implementations (and gRPC is very microservice oriented imho).

I'd be in favor of protobufs, maintaining the definitions in an external library, then wrapping it in Dart and making it a dependency (I'd wrap it in node and make it a dependency too, so that we could sync the changes).

I'm just providing input, I don't really see a negative side in json-rpc either, I just think that protobufs are more standard and require less work to maintain, but feel free to dispute this point (I don't have that much experience and you'll be the one deciding/implementing it first, so choose what you think works best).

if you're unsure just implement a small part of the functionality in a "testing" branch both with json-rpc and protobufs, I'll wrap them in node so that we could have something to base our assumptions on.

TL;DR option 2 seems better

EDIT: it came to my mind later: do we really need bidirectional calls? From my point of view the wrapper just starts dart-sass with the correct options and then doesn't send anything afterwards, it just receives and parses output. Am I missing something?

Sorry for the late reply, been busy studying :)

[nex3]

I'm just providing input, I don't really see a negative side in json-rpc either, I just think that protobufs are more standard and require less work to maintain, but feel free to dispute this point (I don't have that much experience and you'll be the one deciding/implementing it first, so choose what you think works best).

The big downside is the parsing and serialization overhead. The protocol may end up being quite chatty when used to implement functions in the host language.

EDIT: it came to my mind later: do we really need bidirectional calls? From my point of view the wrapper just starts dart-sass with the correct options and then doesn't send anything afterwards, it just receives and parses output. Am I missing something?

The wrapper needs to be able to define functions and importers in the host language that can be called by the Sass implementation. That's where most of the complexity of the protocol will come from.

TL;DR option 2 seems better

I agree. Let's plan on Protobufs + a custom RPC protocol then. I'll create a repo for the protocol definition in the next day or two.

[nex3]

Creating a new repo involves some bureaucracy on the Google end, so it may take a bit, but I should have it up soon.

[ghost]

Any progress?

[nex3]

I'm waiting on one last approval to create the repo.

[nschonni]

There is this https://github.com/Microsoft/language-server-protocol which is used by VS Code for multiple languages and remote debugging between IDEs

[nex3]

Okay, I've created https://github.com/sass/sass-embedded-protocol and sent you an invite to a team with write access. I'll send out a pull request soon with an initial draft of a protocol skeleton.

[ghost]

@nschonni since I feel bad leaving a suggestion without an explicit answer, the point (IMHO) is that we're not trying to communicate with an IDE and we currently don't know what the final spec is going to look like (although Natalie knows more than me for sure), so we may need additional informations (we'll probably do) and/or not use the entirety of the spec (which, to my limited understanding, is just a set of field in a JSON-RPC implementation).

This is a very simple answer, I know, it's hardly accurate and surely I'm missing many things (I gave the spec only a superficial look) so feel free to point out any mistake in my answer.

[nschonni]

no worries, I was just sharing some prior art that might have some overlap

[nex3]

@strychnide now that the protocol is pretty much in place, I think we're ready to start implementing. I've created a repo for you in sass/embedded-host-node, and one for the Dart wrapper in sass/dart-sass-embedded. You're a collaborator in both of them. Do you need anything else to get started?

[ghost]

apart from the fact I really need to check on this issue more often, nope, I don't think I need anything more than that. I've got exams coming tho, so I'll probably start working on it after August, 7th.

Edit: By that date I'd really appreciate having a POC for the compiler part in order to properly test basic functionality.

[nex3]

@strychnide Happy August! Are you about ready to get started?

[ghost]

I volunteered for mantaining the node side of it, but looks like we still need basic functionality in https://github.com/sass/dart-sass-embedded . RIght?

[nex3]

In order for it all to come together, yeah, but I think you can get started without that by just writing tests that fake a super-simple compiler side of the protocol.

[ghost]

sure tomorrow last exam, then I'm on it :)

[nex3]

https://github.com/sass/dart-sass-embedded exists and is in beta releases right now, so I'm going to close this out.

[strarsis]

@nex3: So it is possible now to use dart-sass with Dart VM for higher performance?
How can I enable this? Does this package do this now automatically or do I have to install dart-sass-embedded and pass it somehow to this package?

[nex3]

We're still working on a Node.js wrapper for Embedded Sass in https://github.com/sass/embedded-host-node. We'll post an update on the Sass blog once it's ready for use.