Use workflows for mouse interactions

[mxgrey]

This PR removes the InteractionMode state machine and replaces it with a high-level workflow for switching between mouse interaction services. This high-level workflow uses service injection to allow custom mouse interaction services to be plugged in by downstream users of the site editor.

We introduce several different mouse interaction services that can activated by the high-level workflow. Each of these mouse interaction services is implemented as its own workflow:

• Create edge
• Replace a side of an edge
• Create point
• Replace anchor for a point
• Create path
• Place an object in 2D
• Place an object in 3D

Some other mouse interaction workflows that we should consider adding in the future (left out of this PR since they were not already available or not well supported):

• Add a new anchor into an existing path
• Swap an anchor that a path is using with another anchor
• Re-parent a 3D object to another reference frame

This PR introduces some major changes to 3D object placement for the workcell editor. Placing a 3D object now involves an optional step of selecting a parent reference frame for the object. The workflow goes like this:

1. If the user had something selected when they clicked on the spawn model button, that something will automatically be used as the parent.
2. If nothing was selected when the spawn model button was clicked, we will not start out with any parent.
3. Pressing Esc while a parent is selected will clear out the parent, leaving the workflow state similar to (2).
4. Clicking on an object in the scene while no parent is selected will select that object as the parent.
5. Clicking on a spot on the selected parent will place the object at the intersecting location on the parent, with the z-axis aligned with the normal vector of the chosen spot.
6. Clicking somewhere that's not on the parent object will place the object in the x-y plane of the parent's frame.
7. Clicking somewhere that's not on any object will place the object there, whether or not a parent is selected.
8. Holding shift and clicking will ignore any hovering that is happening, placing the object in the x-y plane of the parent frame, or in the x-y plane of the world if no parent is selected.

All of this is communicated to the user through a combination of cursor-attached previewing and a new cursor-attached tooltip feature.

This PR also introduces some crucial fixes to model loading which had some reliability issues around potentially random system ordering and incomplete property propagation.

[luca-della-vedova]

Some other mouse interaction workflows that we should consider adding in the future (left out of this PR since they were not already available or not well supported):

• Re-parent a 3D object to another reference frame

This feature is necessary for workcell calibration, the sequence is usually the following:

• Spawn a mesh in the desired position, its parent can be anything.
• Spawn a calibration frame for the mesh with the intended parent of the mesh. Snap it to a corner of the mesh where a calibration datum is placed.
• Reparent the mesh to be a child of the calibration frame. Note that reparenting calculates the Transform such that the GlobalTransform would be unchanged, so the position of the mesh will be the same.
• Run the calibration routine which readjusts the calibration frames based on external input.
• The mesh position will be automatically adjusted as well since it is a child of the calibration frame.

If we cannot reparent a mesh to a different anchor we can't do this calibration workflow, so the functionality is needed.

The most important point however, is in the redesign of the parenting UX.

It is a deliberate design choice in the current workcell editor that the only entity that can be a parent of another entity is a Frame. This is necessary for a few reasons:

• Allowing arbitrary hierarchies becomes messy very easily, for example a model that is a child of another model, then has a frame as a child, how do we translate this hierarchy to a robot with links and joints? Only having frames as parents makes this easier.
• URDF interoperability, one of the most important features of the workcell editor is to export it as a URDF, as well as being able to have a "tf tree" that is compatible with ROS 2. This works if we keep the concept of frames only being children of other frames. We simply translate Frame to TF frames and URDF links, meshes as visual and collision elements of links, Inertia children of Frames as their inertial components. However once we let frames be children of meshes or meshes be children of other meshes this conversion becomes a lot harder (if even possible?).

I'm attaching a videos of a few of the rough edges that came up while allowing models to be parents of other entities:
Screencast from 2024-08-22 12-06-54.webm

For example when spawning a model as a child of a model the model disappears, frames that are children of models seem to be embedded in the model when saving and reloading. TBH I would just only allow frames as parents to avoid all of these and make the URDF interoperability easier.

Additionally, there was a check in place to make sure we only do the "mesh snapping" when hovering over models. This is necessary to avoid snapping to the anchor meshes as the video below (which is probably not something that a normal user would want):

Screencast.from.2024-08-22.12-19-04.webm

I need to setup the rest of the infrastructure to test the rest of the workcell editor pipeline, but these are the points I could find immediately

Edit: Ah I just noticed I didn't capture the mouse pointer :| Apologies for that! I hope the videos still make sense

[mxgrey]

If we cannot reparent a mesh to a different anchor we can't do this calibration workflow, so the functionality is needed.

I see, the only reason I didn't include it in this PR is because I wasn't sure how to preview it to the user. I thought we'd want to snap the model back to the cursor while selecting the new parent, and that would've been very difficult to propagate properties correctly.

What you've described, where the object remains at the same global transform while we reparent, will be very easy to implement, so I'll go ahead and add that in now.

It is a deliberate design choice in the current workcell editor that the only entity that can be a parent of another entity is a Frame.

This should be very easy to fix. I just need to drill up to the first ancestor frame of the selected parent before inserting the object into the world. I think down the road we should consider whether it's possible to automatically consider every tangible object in the scene to be a frame, but that's obviously outside the scope of this PR.

[mxgrey]

• e9a36cf makes sure that only frames can be parents to objects (anchors, models, visuals, and collisions)
• fdcd8c0 prevents the intersection ray from hitting the round surface of anchors
• 49ab9b9 brings back the reparenting selection mode. Since there's no preview while using this mode, I found it difficult as a user to tell whether the app is in "select parent" mode versus being in regular inspector mode. To mitigate that problem, I added a tooltip that's always on during this mode to indicate that you're in parent selection mode.

The vanishing model issue was fixed with e9a36cf. I also tested a round-trip of saving and reloading, and everything appeared to be working: everything loaded back up, and all the parent-child relationships appeared to be in tact.

I believe this addresses all the current feedback, but please let me know if I missed anything.

[luca-della-vedova]

It's a lot to go through, I left a few minor feedback around and rough edges during my testing.

It's probably quite clear given that the only real code feedback I gave is on a small isolated module that this change is very complex to understand and I admittedly don't understand most of it. I think what is most valuable is trying to document as much as possible the complex workflows that are introduced, what their steps do, how users are supposed to configure them and a bit more explanation on the actual complex workflow APIs used.
Imo this is important especially since bevy_impulse is in very early stages with very little applications. This is pretty much the first complex application and will probably be referenced a lot by people when trying to design their own workflows.

I'll also have to understand this in quite a lot of detail when trying to work on the workcell editor mode but I suspect that will take some weeks and it's probably not worth it to keep this PR in limbo until then.

[luca-della-vedova]

It would be helpful to have some documentation on what this function is meant to achieve and what the steps do, otherwise a quite deep knowledge of bevy_impulse will be needed to decode it

[mxgrey]

Explanation of the workflow as well as a breakdown of the operations added here. This one would probably be much easier to understand with a graphical representation. I might try to poke around for a suitable tool for creating and rendering DOT files that we might link to in the docs.

[luca-della-vedova]

This is also a fairly complex function, maybe a breakdown of what each service that users are expected to set is supposed to do would help?

[mxgrey]

Service expectations outlined here.

[luca-della-vedova]

Last nit is that this PR introduces a bunch of extra warnings, like unused variables and imports that increase noise and make it more likely that we will miss important ones in the future. Should make sure to clean them up before merging

[mxgrey]

Ah yeah actually quite a few warnings that have been lingering were kept on purpose as reminders for issues that need to be addressed in the old mouse selection state machine, so this is a good opportunity to clean up any remaining warnings.

But I don't look forward to the day we try to apply clippy to this project...

[mxgrey]

I think all the latest feedback is addressed.

I also realized that I accidentally got rid of the ability to deselect the current inspected item by pressing the Esc key so I added that back in with ca82bee.

[luca-della-vedova]

Thanks for iterating!