(perception_benchmark_tool): add package

[kaancolak]

Signed-off-by: Kaan Colak kcolak@leodrive.ai

Description

Resolves #565
Builds on top of #565

Related links

Tests performed

Notes for reviewers

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

• I've confirmed the contribution guidelines.
• The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

• The PR follows the pull request guidelines.
• The PR has been properly tested.
• The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

• There are no open discussions or they are tracked via tickets.
• The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

[kaancolak]

@ktro2828 Could you review this package?

We've decided to exclude it from Autoware.Universe. The dependencies of waymo-open-dataset-tf-2-6-0 aren't written properly.

If install it directly from pip with other dependencies, everything works.

[ktro2828]

@kaancolak
Hmm, in my case, the process was stucked with Autoware ready log as below.

$ ros2 launch perception_benchmark_tool benchmark_runner.launch.py dataset_path:=/media/kotarouetake/Volume/DATASET/waymo/validation/
[INFO] [launch]: All log files can be found below /home/kotarouetake/.ros/log/2022-10-12-13-23-23-473536-dpc1804002-767353
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [benchmark_node-1]: process started with pid [767355]
[INFO] [waymo_player_node-2]: process started with pid [767357]
[INFO] [autoware_workflow_runner_node-3]: process started with pid [767359]
[waymo_player_node-2] [INFO 1665548606.541920890] [waymo_player_node]: Waymo segment decoding from dataset...
[waymo_player_node-2] 2022-10-12 13:23:26.581654: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.602333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.602494: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.603136: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
[waymo_player_node-2] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[waymo_player_node-2] 2022-10-12 13:23:26.604096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.604240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.604380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.964697: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.964879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.965013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.965125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8736 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
[waymo_player_node-2] 2022-10-12 13:23:26.987997: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
[waymo_player_node-2] 2022-10-12 13:23:42.104813: I tensorflow/core/util/cuda_solvers.cc:180] Creating CudaSolver handles for stream 0x6caf2a0
[waymo_player_node-2] [INFO 1665551682.191315314] [waymo_player_node]: Waymo segment decoded
[autoware_workflow_runner_node-3] [INFO 1665551684.211187153] [autoware_workflow_runner_node]: Waiting for Autoware's subscriber to be ready
[autoware_workflow_runner_node-3] [INFO 1665551686.196037136] [autoware_workflow_runner_node]: Waiting for Autoware's subscriber to be ready
[autoware_workflow_runner_node-3] [INFO 1665551688.196780713] [autoware_workflow_runner_node]: Waiting for Autoware's subscriber to be ready
[autoware_workflow_runner_node-3] [INFO 1665551688.198051545] [autoware_workflow_runner_node]: Autoware ready.

What I did after building autoware and perception_benchmark_tool

$ cd ~/autoware
$ source install/setup.bash
$ cd ~/perception_benchmark_tool_ws/src
$ source install/setup.bash

$ ros2 launch perception_benchmark_tool benchmark_runner.launch.py dataset_path:=MY_WAYMO_PATH

dataset configuration is following

MY_WAYMO_PATH/
├── LICENSE
├── xxx.tfrecord
...

I downloaded validation_0000.tar from here

[kaancolak]

@kaancolak Hmm, in my case, the process was stucked with Autoware ready log as below.

$ ros2 launch perception_benchmark_tool benchmark_runner.launch.py dataset_path:=/media/kotarouetake/Volume/DATASET/waymo/validation/
[INFO] [launch]: All log files can be found below /home/kotarouetake/.ros/log/2022-10-12-13-23-23-473536-dpc1804002-767353
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [benchmark_node-1]: process started with pid [767355]
[INFO] [waymo_player_node-2]: process started with pid [767357]
[INFO] [autoware_workflow_runner_node-3]: process started with pid [767359]
[waymo_player_node-2] [INFO 1665548606.541920890] [waymo_player_node]: Waymo segment decoding from dataset...
[waymo_player_node-2] 2022-10-12 13:23:26.581654: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.602333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.602494: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.603136: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
[waymo_player_node-2] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[waymo_player_node-2] 2022-10-12 13:23:26.604096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.604240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.604380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.964697: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.964879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.965013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[waymo_player_node-2] 2022-10-12 13:23:26.965125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8736 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1
[waymo_player_node-2] 2022-10-12 13:23:26.987997: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
[waymo_player_node-2] 2022-10-12 13:23:42.104813: I tensorflow/core/util/cuda_solvers.cc:180] Creating CudaSolver handles for stream 0x6caf2a0
[waymo_player_node-2] [INFO 1665551682.191315314] [waymo_player_node]: Waymo segment decoded
[autoware_workflow_runner_node-3] [INFO 1665551684.211187153] [autoware_workflow_runner_node]: Waiting for Autoware's subscriber to be ready
[autoware_workflow_runner_node-3] [INFO 1665551686.196037136] [autoware_workflow_runner_node]: Waiting for Autoware's subscriber to be ready
[autoware_workflow_runner_node-3] [INFO 1665551688.196780713] [autoware_workflow_runner_node]: Waiting for Autoware's subscriber to be ready
[autoware_workflow_runner_node-3] [INFO 1665551688.198051545] [autoware_workflow_runner_node]: Autoware ready.

What I did after building autoware and perception_benchmark_tool

$ cd ~/autoware
$ source install/setup.bash
$ cd ~/perception_benchmark_tool_ws/src
$ source install/setup.bash

$ ros2 launch perception_benchmark_tool benchmark_runner.launch.py dataset_path:=MY_WAYMO_PATH

dataset configuration is following

MY_WAYMO_PATH/
├── LICENSE
├── xxx.tfrecord
...

I downloaded validation_0000.tar from here

@ktro2828 is RVIZ2 opened and just waiting?

According to this log "[autoware_workflow_runner_node]: Autoware ready." autoware initialized without a problem.

I am trying to investigate the problem but haven't reproduced it.

[ktro2828]

@kaancolak

is RVIZ2 opened and just waiting?

Yes, rviz2 is opend and just waiting like this.

[kaancolak]

@ktro2828 In my workstation and desktop everything works without any problem. But today I tested with my colleague's computer and faced the same problem.

The cause of the problem seems to be Cyclone DDS. Autoware does not work properly with Cyclone's default settings. So many messages are dropping with the default configuration. After tuning DDS, everything worked.

Did you tune your Cyclone DDS?

ROS2 Cyclone DDS tuning

For a better configuration file, you can use provided config file under this topic ;
https://github.com/orgs/autowarefoundation/discussions/2636

[ktro2828]

Thanks, It worked! LGTM

[epris]

Dear @xmfcx @kaancolak ,
I'm sorry to comment on this old post but I have a problem with benchmark_node.py
It calls a callback when ground truth messages and perception messages have the same timestamp
self.sub_tracking = Subscriber(self, TrackedObjects, "/perception/object_recognition/tracking/objects") self.sub_tracking_gt = Subscriber(self, TrackedObjects, "/gt_objects") self.time_sync = TimeSynchronizer([self.sub_tracking, self.sub_tracking_gt], 10) self.time_sync.registerCallback(self.tracking_benchmark_callback)
The problem is that it never happens. In fact, the two sets of messages have different time stamps.

For example:
here I'm printing tracked_objects.header.stamp in tracked_objects_callback
[autoware_workflow_runner_node-3] [INFO] [1718807858.002042033] [autoware_workflow_runner_node]: builtin_interfaces.msg.Time(sec=1718807857, nanosec=999909839)

here i'm printing ground_truth_objects.header.stamp = ros_time.to_msg() in publish_gt_objects
[waymo_player_node-2] [INFO] [1718807858.011823941] [waymo_player_node]: builtin_interfaces.msg.Time(sec=1553735873, nanosec=137396000)

What could be the issue in my case?