Hot reloading of Rust functions for mods and faster development

[alice-i-cecile]

What problem does this solve or what need does it fill?

1. Powerful mods (think Factorio or Minecraft) may want to modify the game's core code directly.
2. As games grow in size, testing changes becomes increasingly onerous. You need to both recompile and recreate the relevant state in order to see if your change fixed the bug or has the desired gameplay impact. This slows down development velocity significantly, and pushes more and more behaviour into hacky secondary systems (like in-game consoles or dev-specific UI elements) in order to ensure effective testing.

Describe the solution would you like?

Load in new function definitions from shared object files, allowing changes to game logic while it is running.

The basic behaviour outlined in the hotpatch crate works smoothly for mods, but for development, it would be ideal to have it automatically detect changes to the saved source files (when an appropriate flag is set). This behaviour is very similar to the existing hot reloading of assets.

A few caveats:

1. Hot reloading relies on shared objects, defined by the ABI. But ABI stability is not guaranteed across Rust compiler versions though, which will need to be carefully considered and documented for this feature. The hotpatch crate recommends using the C ABI for stability, although I'm not knowledgeable enough to speak on the full implications of this choice or feasible alternatives.
2. The ability to modify the underlying engine code might also pose serious security issues for some games, so this should be an opt-in feature, typically turned off for releases.
3. The limitation on not modifying the current function or its parents might be hard to accomplish in ECS given the continually running game loop. As changes to systems seem much more plausible than changes to the underlying scheduler or wrapper logic, you'd likely want to queue up changes until the start of the next time-step (or stage?) to be sure that you're not accidentally replacing a system that's currently executing.
4. The need to annotate functions with #[patchable] would get old quickly: we'd need to find some refactoring to replace this requirement with a global feature flag.

Describe the alternative(s) you've considered?

Mods could work by purely adding code, integrating a scripting language, or incorporating this sort of functionality in an external library. The potential benefits to development speed push me towards offering this as a core bit of functionality if possible.

Development / testing loops might be fast enough with dynamic linking to improve compile times + high quality state saving and loading. Without testing on large code bases, it's unclear what sort of iteration speeds this might offer.

Additional context

The new hotpatch crate, discussed on reddit here was the inspiration for this feature request, and seems to have some nice technical capabilities. As is, we can't incorporate it due to the GPL license, but we could create an MIT re-implementation or work with its author to change the license.

[ambeeeeee]

I have one comment

but we could create an MIT re-implementation or work with its author to change the license.

If you've looked at the code at all, even for a moment, you risk a "re-implementation" being deemed "derivative work" and thus also having to be licensed under the GPL. The other option is feasible, but you'd have to get permission from all three contributors to do that. This isn't impossible though so its worth trying if this is something important to bevy.

[Shizcow]

hotpatch dev here. I am willing and able to change the license. The crate is now under MIT+Apache as is standard.

This crate is also really new, only on it's first published version. Development will be active again shortly, so if the bevy team needs additional features or fixes that's an option.

One comments on OP's message:

The limitation on not modifying the current function or is parents ...

This currently is supported in hotpatch, but is unsafe. As I understand the implications of modifying global statics without multithreading protection more, documentation will expand to include exactly what should and shouldn't be done.

[alice-i-cecile]

This blog post looks to be a useful log of exploring some of the implementation details of this sort of work. It's very detailed and stream-of-thought though, so it's probably more useful once we actually start building.

[zicklag]

Development / testing loops might be fast enough with dynamic linking to improve compile times + high quality state saving and loading. Without testing on large code bases, it's unclear what sort of iteration speeds this might offer.

That's my initial thought: that with dynamically linked game code that actually arleady have the build times might not even get that bad for small-medium games. But also it seems like the concept of hot-reloaded systems may be rather simple to accomplish if you only patched them at the end of every frame. ( I'm no expert, though. :) )

[rookboom]

I would advocate for using WASM instead of hotloading DLLs. In addition to allowing hotloading during development, WASM modules run in a sandbox environment and can only interact with the host in a predefined way. Thus they can be trusted not to execute malicious code. This opens up the possibility for user generated code extensions for Bevy games and also the Bevy editor, (once we have one). Microsoft did this in the latest Flight Simulator to enable safe community extensions.

[bjorn3]

The disadvantage of using wasm is that it requires writing glue code for every type that needs to be passed between the wasm plugin and the main game executable. Also sometimes this sandboxing may not be desired. For example when requiring direct access to os libraries. Finally it requires shipping a wasm compiler, which may not be desired for code size reasons or because the target os doesn't allow jitting. cough iOS cough Supporting both WASM and dynamic libraries would be an option though I think. Allowing to choose the most appropriate tool for the job.

[alice-i-cecile]

An interesting article (and associated crate) exploring this for use with Bevy: https://robert.kra.hn/posts/hot-reloading-rust/

Some very serious limitations, but may still be useful if you build a workflow around it.