Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting Using 2D Masks

This repository contains the code for the paper Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting Using 2D Masks.

Project page: https://jojijoseph.github.io/3dgs-segmentation Preprint: https://arxiv.org/abs/2409.11681

Update: Sep 24, 2024 - Our poster titled Segmentation of 3D Gaussians using Masked Gradients, corresponds to the preliminary work, has been accepted to SIGGRAPH Asia 2024.

Setup

Please install the dependencies listed in requirements.txt via pip install -r requirements.txt. Download sam2_hiera_large.pt from https://huggingface.co/facebook/sam2-hiera-large/tree/main and place it in the ./checkpoints folder.

Other than that, it's a self-contained repo. Please feel free to raise an issue if you face any problems while running the code.

Demo

python demo.py --help

If needed, sample data (chair) can be found here. Please create a folder named data on root folder and extract the contents of zip file to that folder. Then simply run python demo.py.

To see the output after segmentation,

python viewer.py

Please type python viewer.py --help to see more options.

demo.mp4

Affordance Transfer

python affordance_transfer_pipeline.py --help

Left: Source images, Middle: 2D-2D affordance transfer, Right: 2D-3D Affordance transfer

affordance_transfer.mp4

Affordance Transfer - Evaluation

Download trained scenes from here. Original scenes (without trained Gaussian Splat models) can be found at https://users.umiacs.umd.edu/~fer/affordance/Affordance.html.

sh eval_affordance_transfer.sh | tee affordance_transfer.log

Some Downstream Applications

Augmented reality.

ar_example.mp4

Reorganzing objects in real time.

move_gaussian.mp4

Acknowledgements

A big thanks to the following tools/libraries, which were instrumental in this project:

• gsplat: 3DGS rasterizer.
• SAM 2: To track masks throughout the frames.
• YOLO-World via ultralytics: To find Initial bounding box.
• labelme: To label the source images for affordance transfer.

Citation

If you find this paper or the code helpful for your work, please consider citing our preprint,

@article{joji2024gradient,
  title={Gradient-Driven 3D Segmentation and Affordance Transfer in Gaussian Splatting from 2D Masks},
  author={Joji Joseph and Bharadwaj Amrutur and Shalabh Bhatnagar},
  journal={arXiv preprint arXiv:2409.11681},
  year={2024},
  url={https://arxiv.org/abs/2409.11681}
}