Antoine Guédon  Vincent Lepetit 
LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS
| Presentation video | SuGaR web page | Frosting web page |

Using our methods SuGaR (CVPR 2024) or Gaussian Frosting (ECCV 2024), you can reconstruct editable meshes from RGB images or RGB videos, and render them using realistic Gaussian Splatting rendering.
We now provide a Blender Add-On to edit, sculpt, combine or animate 3D scenes reconstructed with SuGaR of Frosting and render them with Gaussian Splatting without a single line of code.

Installation

1. Please start by cloning this repo.
   

2. Then, install Blender (version 4.0.2 is recommended but not mandatory).
   

3. Open Blender, and go to Edit > Preferences > Add-ons > Install, and select the file sugar_addon.py located in this repo.
   

You have now installed the SuGaR x Frosting addon for Blender!

Usage

1. Please start by installing either SuGaR or Gaussian Frosting from the respective repositories.

2. Follow the instructions from the repositories to optimize a SuGaR or Frosting model.

3. Open a new scene in Blender, and go to the Render tab in the Properties. You should see a panel named Add SuGaR or Frosting mesh. The panel is not necessary at the top of the tab, so you may need to scroll down to find it.
   
   

Use the `Add SuGaR or Frosting mesh` panel to load a mesh
reconstructed with SuGaR or Frosting in Blender.

4. (a) Select a mesh. Enter the path to the final mesh extracted from SuGaR or Frosting in the Path to OBJ file field. You can also click on the folder icon to select the file. The mesh should be located in SuGaR/output/refined_mesh/<YOUR SCENE NAME> or in Frosting/output/refined_frosting_base_mesh/<YOUR SCENE NAME> depending on the model you used.
   
   (b) Select a checkpoint. Similarly, enter the path to the final checkpoint of the optimization in the Path to PT file field. You can also click on the folder icon to select the file. The checkpoint should be located in SuGaR/output/refined/<YOUR SCENE NAME> or in Frosting/output/refined_frosting/<YOUR SCENE NAME> depending on the model you used.
   
   (c) Load the mesh. Finally, click on Add mesh to load the mesh in Blender. Feel free to rotate the mesh and change the shading mode to better visualize the mesh and its colors.
   
   

Please rotate the mesh and change the shading mode to get a better view of the mesh.

5. Now, feel free to edit your mesh using Blender!
   You can segment it into different pieces, sculpt it, rig it, animate it using a parent armature, etc. You can also add other SuGaR or Frosting meshes to the scene, and combine elements from different scenes.
   Feel free to set a camera in the scene and prepare an animation: You can animate the camera, the mesh, etc.
   Please avoid using Apply Location, Apply Rotation, or Apply Scale on the edited mesh, as I am still not sure how it will affect the correspondence between the mesh and the optimized checkpoint.
   
   

You can edit the mesh as you want, and prepare an animation with it.
You can also add other SuGaR or Frosting meshes to the scene.

6. Once you're done with your editing, you can prepare a rendering package ready to be rendered with SuGaR or Frosting. To do so, go to the Render tab in the Properties again, and select the main directory of your model in the Path to SuGaR/Frosting directory field. If you installed the repo of SuGaR or Frosting from GitHub, this directory should be named either SuGaR or Frosting.
   Finally, click on Render Image or Render Animation to render the scene.
   
   

In this example, we first animated the camera and the character.
Then, we added another Frosting mesh to get a background for our character.

Render Image will render a single image of the scene, with the current camera position and mesh editions/poses.

Render Animation will render a full animation of the scene, from the first frame to the last frame you set in the Blender Timeline.

The package should be located in <SuGaR or Frosting>/blender/packages/.

7. Finally, you can render the package with SuGaR or Frosting. You just need to go to the root directory of the SuGaR or Frosting repo and run the following command:

python render_blender_scene.py -p <PATH TO PACKAGE>

With SuGaR or Frosting, you can get a realistic, high-quality rendering of your scene!

Please check the documentation of SuGaR or Frosting and the render_blender_scene.py scripts for more information on the additional arguments. If you get artifacts in the rendering, you can try to switch the automatic adjustment method of the Gaussians from 'complex' to 'simple', as the simple method is less accurate but faster and more robust to extreme deformations:

python render_blender_scene.py -p <PATH TO PACKAGE> --adaptation_method simple

If you notice that some Gaussians are not rendered (especially if you perform extreme deformations on the meshes), you can also change the --deformation_threshold argument. All Gaussians belonging to triangles with a deformation factor higher than this threshold will be culled during rendering. The default value is 2., but you can try to increase it to 5. or 10:

python render_blender_scene.py -p <PATH TO PACKAGE> --deformation_threshold 10

8. Instead of rendering the frames, you can also export a PLY file of the Frosting representation at a specific frame. This PLY file can be used to visualize the Frosting representation in any 3D Gaussian Splatting viewer, such as SuperSplat. To do so, add the argument --export_frame_as_ply followed by the frame number you want to export. For example, for exporting a PLY file of the representation at frame 10:

python render_blender_scene.py -p <PATH TO PACKAGE> --export_frame_as_ply 10

You can visualize the edited SuGaR or Frosting representation at a specific frame
in any 3D Gaussian Splatting viewer, such as SuperSplat!