Shell Reconstruction

This repository provides the source code, trained model and instructions for using shell reconstruction.

Shell reconstruction is a novel method for 3D shape reconstruction which models the object geometry as a pair of depth images, composing the "shell" of the object. This unique representation allows using image-to-image residual ConvNet architectures for 3D reconstruction, generates object reconstruction directly in the camera frame, and generalizes well to novel object types. Moreover, an object shell can be converted into an object mesh in a fraction of a second, providing time and memory efficient alternative to voxel or implicit representations. Please visit our arxiv paper for more details.

Setup

• Install git-lfs
• Clone the repository
• Setup python environment as following

conda create --prefix ./pyvenv python=3.9
conda activate ./pyvenv
pip install --upgrade pip
pip install -r requirements.txt

Usage API

Given a masked depth image of the target object and the camera intrinsics matrix, shell reconstruction can be invoked in the following way:

from shell.reconstructor import ShellReconstructor
reconstructor = ShellReconstructor()
np_pcd = reconstructor.reconstruct(masked_depth, camera_k)

Please look at the reconstructor file for more details on input parameters and output.

Demo

We provide real-world scene information (demo_data/real_data.pkl) captured using Intel RealSense RGBD camera. You can see shell-reconstruction demo on this data by running:

python demo.py

This file will generate reconstructions in demo_data/reconstruction folder (we also provide expected output in demo_data/reconstruction_expected folder).

ROS2 Node:

For running shell-reconstruction as a ROS2 node, please have a look at shell-reconstruction-ros repository.

Citation

@article{chavan2021shellgrasp,
  title={Simultaneous Object Reconstruction and Grasp Prediction using a Camera-centric Object Shell Representation},
  author={Chavan-Dafle, Nikhil and Popovych, Sergiy and Agrawal, Shubham and Lee, Daniel D and Isler, Volkan},
  journal={arXiv preprint arXiv:2109.06837},
  year={2022}
}

Acknowledgment

• Colin Prepscius for extensive code review