How to generate ground truth color and depth images from Replica dataset for 3D reconstruction?

[sanbuddhacharyas]

I followed the tutorial from https://github.com/facebookresearch/habitat-sim/blob/main/examples/tutorials/colabs/ECCV_2020_Navigation.ipynb and used "observations = sim.step("turn_left")" to generate the depth and colour image of the 3D scene but, the output images are deformed and there is radial and tangential distortion in the images and when I reconstruct 3d model using open3d, the output is very bad. How can I get the real ground truth color and depth images so that I can reconstruct a 3D model from the captured data.

agent and simulator configuration
simulator backend
sim_cfg = habitat_sim.SimulatorConfiguration()
sim_cfg.scene_id = settings["scene"]

In the 1st example, we attach only one sensor,

a RGB visual sensor, to the agent

rgb_sensor_spec = habitat_sim.CameraSensorSpec()
rgb_sensor_spec.uuid = "color_sensor"
rgb_sensor_spec.sensor_type = habitat_sim.SensorType.COLOR
rgb_sensor_spec.resolution = [settings["height"], settings["width"]]
rgb_sensor_spec.position = [0.0, settings["sensor_height"], 0.0]
rgb_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE

depth_sensor_spec = habitat_sim.CameraSensorSpec()
depth_sensor_spec.uuid = "depth_sensor"
depth_sensor_spec.sensor_type = habitat_sim.SensorType.DEPTH
depth_sensor_spec.resolution = [settings['height'], settings['width']]
depth_sensor_spec.position = [0.0, settings['sensor_height'], 0.0]
depth_sensor_spec.sensor_subtype = habitat_sim.SensorSubType.PINHOLE

agent

agent_cfg = habitat_sim.agent.AgentConfiguration()
agent_cfg.sensor_specifications = [rgb_sensor_spec, depth_sensor_spec]
agent_cfg.action_space = {
"move_forward" : habitat_sim.agent.ActionSpec("move_forward", habitat_sim.agent.ActuationSpec(amount=0.1)),
"turn_left" : habitat_sim.agent.ActionSpec("turn_left", habitat_sim.agent.ActuationSpec(amount=1.0)),
"turn_right" : habitat_sim.agent.ActionSpec("turn_right", habitat_sim.agent.ActuationSpec(amount=1.0))
}

[erikwijmans]

EDIT: Saw this first before the dup on habitat-sim's GitHub repo. Since this is a habitat-sim question, not a replica question, I'll favor that issue for future discussion: facebookresearch/habitat-sim#1732