This repository contains the implementation of the paper:
Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling"
Xinhang Liu, Yu-Wing Tai, Chi-Keung Tang, Pedro Miraldo, Suhas Lohit, Moitreya Chatterjee
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 (Highlight)
To make free viewpoint rendering systems ubiquitous, it is essential that they be capable of handling dynamic scenes, i.e. those where objects change in their position or configuration or both over time. Existing approaches for this task, propose pipelines that are agnostic to the semantic content of the scene and thus treat every region in the 3D space, as being equally important, when rendering. This results in the system struggling to render the regions of the scene that have high motion. In this paper, we depart from such a simplistic rendering pipeline by adjusting the spatio-temporal sampling resolution of the different semantic regions of the scene, based on the extent of their motion. These regions are grouped based on this criterion and each such region is called a "Gear". We propose to sample more densely from regions with high motion, i.e. those that are assigned higher gears. This results in noticeable improvement in rendering quality over the state-of-the-art approaches, across a wide variety of dynamic scenes. Furthermore, almost for free, our proposed method enables free-viewpoint tracking of objects of interest starting from a single mouse click - a functionality not yet achieved by prior methods.
To install all required python dependences run
conda env create -f environment.yml
conda activate gearnerf
sh Install.sh
Download the SAM-HQ checkpoint and put it in the ~/pre_trained folder.
Note that we tested the Gear-NeRF codebase on a machine running Ubuntu 22.04, with an NVIDIA 3090 RTX GPU, CUDA version 11.6, and 128 GB of RAM.
By default, we assume that:
- All datasets are located in the
~/datafolder (specified by theexperiment.params.data_dirargument) E.g. data/immersive/05_Horse/camera_0001.mp4 - With the subdirectory for each individual dataset specified by the
experiment.dataset.data_subdirargument (e.g., seeconf/experiment/params/local.yamlandconf/experiment/dataset/technicolor.yaml).
Download the Google Immersive sequences from their release page. As an example, in order to download the flames sequence, run:
wget https://storage.googleapis.com/deepview_video_raw_data/02_Flames.zip
Then extract the sequence to the ~/data folder as stated above.
By default...
- Checkpoints are written to the
~/checkpointsfolder (specified by theexperiment.params.ckpt_dirargument) - Logs are written to the
~/logsfolder (specified bt theexperiment.params.log_dirargument).
In order to train Gear-NeRF on the Horse scene from the Google Immersive dataset, run:
sh scripts/train_horse.sh <gpu_to_use>
After training, you should see the rendered RGB image as well as SAM feature maps with visualization. These feature maps can be turned into segmentation masks according to various user inputs.
To render a video of RGB and SAM feature maps with an automatically generated camera trajectory, run:
sh scripts/render_horse.sh <gpu_to_use>
To render with a given camera pose and time step specified in scripts/input_pose.json, run:
sh scripts/render_horse_given_pose.sh <gpu_to_use>
In order to track a region in novel views, run the following command:
python demo_tracking.py
One needs to provide the coordinates of where the user would have clicked on a given frame in Line 108 of demo_tracking.py.
See CONTRIBUTING.md for our policy on contributions.
The code of Gear-NeRF is inspired from: HyperReel, which in turn has been inspired from: nerf_pl, TensoRF, torch-ngp. We are also grateful to authors of the following repositories: SAM, and SAM-HQ
@inproceedings{liu2024gear,
title={Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling},
author={Liu, Xinhang and Tai, Yu-Wing and Tang, Chi-Keung and Miraldo, Pedro and Lohit, Suhas and Chatterjee, Moitreya},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={19667--19679},
year={2024}
}
Released under AGPL-3.0-or-later license, as found in the LICENSE.md file.
All files, except as noted below:
Copyright (c) 2024 Mitsubishi Electric Research Laboratories (MERL)
SPDX-License-Identifier: AGPL-3.0-or-later
The files under the folders:
~/datasets/~/scripts/
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
The files under the folder:
~/utils/
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
Copyright (c) 2022 hawkey
except the following files:
~/utils/sh_utils.py~/utils/tensorf_utils.py
which were also taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
Copyright (c) 2022 Anpei Chen
The files under the folder:
~/conf/
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
Copyright (c) 2020 Quei-An Chen
The files under the folder:
~/nlf/embedding~/nlf/intersect~/nlf/models~/nlf/regularizers~/nlf/visualizers
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
The files under the folder:
~/nlf/conf/
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
Copyright (c) 2020 Quei-An Chen
The following files:
~/nlf/nets/array_nd.py~/nlf/nets/__init__.py~/nlf/nets/mlp.py~/nlf/nets/nets.py~/nlf/nets/siren.py~/nlf/nets/tensorf.py~/nlf/nets/tensorf_density.py~/nlf/nets/tensorf_hybrid.py~/nlf/nets/tensorf_no_sample.py~/nlf/nets/tensorf_reflect.py
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
Copyright (c) 2022 Anpei Chen
The following files:
~/nlf/activations.py~/nlf/contract.py~/nlf/param.py~/nlf/pe.py~/nlf/rendering.py~/nlf/subdivision.py
were taken without modification from here (license included in LICENSES/MIT.txt), with the following copyrights:
Copyright (c) Meta Platforms, Inc. and affiliates.
The following files:
~/nlf/__init__.py~/losses.py~/main.py~/metrics.py~/environment.yml
were adapted from here (license included in LICENSES/MIT.txt), with the following copyrights:
# Copyright (c) 2024 Mitsubishi Electric Research Laboratories (MERL)
# Copyright (c) Meta Platforms, Inc. and affiliates.