Move downloading artifacts outside CMake

[esteve]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I've agreed with the maintainers that I can plan this task.

Description

Some packages download artifacts by default, which is not necessary for compiling them. Additionally, during the process for generating Debian packages, access to the network is restricted, so building these packages fail (see autowarefoundation/autoware#3222 (comment))

The packages that download ONNX models are:

• build(lidar_apollo_instance_segmentation): remove download logic from CMake #3140
• build(lidar_centerpoint): remove download logic from CMake #3141
• build(tensorrt_yolo): remove download logic from CMake #3142
• build(tensorrt_yolox): remove download logic from CMake #3143
• build(traffic_light_classifier): remove download logic from CMake #3144
• build(traffic_light_ssd_fine_detector): remove download logic from CMake #3145
• build(traffic_light_fine_detector): remove download logic from CMake #4896
• build(image_projection_based_fusion): remove downloading logic from Cmake #4904
• build(yabloc_pose_initializer): remove downloading logic from CMake #4905

Other types of artifacts:

• fix(yabloc_pose_initializer): disable downloading artifacts by default #4110
• feat(ansible): add yabloc archive extraction to andsible script autoware#3855
• build(tvm_utility): remove download logic from CMake and update documentation #4923
• build(lidar_centerpoint_tvm): remove artifacts download #5367
• build(lidar_apollo_segmentation_tvm and lidar_apollo_segmentation_tvm_nodes): remove download from cmake #5431

Update ansible scripts to download artifacts files:

• feat: use ansible to download ONNX files autoware#3375
• fix: add updated artifacts for tensorrt_yolox autoware#3848
• fix: update all downloaded files with their SHA256 checksum autoware#3856
• fix: update all downloaded files with their SHA256 checksum (2) autoware#3866
• feat(ansible): tvm artifacts download autoware#3916

Update documentation with instructions on how to download ONNX files:

• docs: add documentation about ONNX models autoware-documentation#345
• docs: add updated artifacts for tensorrt_yolox autoware-documentation#456
• docs(ansible): add readme with manual artifacts downloading explanation autoware#3850
• docs(installation): add link to manual artifacts downloading instruction autoware-documentation#462
• docs(tutorials): add link to manual artifacts downloading autoware-documentation#463

Disable internet access for CI actions:

• build: disable internet access during the build step autoware-github-actions#241

Add data_path variable to autoware_launch:

• feat(perception): add data_path argument to launch file autoware_launch#577
• feat(perception_launch): add data_path arg to perception launch #5069

Update default path to artifacts in package launch files:

• fix(traffic_light_fine_detector): add data_path arg and update default model path #5118
• fix(tensorrt_yolox): add data_path arg and update default model path #5119
• fix(lidar_apollo_instance_segmentation): add arg data_path, update model path #5139
• fix(traffic_light_ssd_fine_detector): add arg data_path, update model file path #5141
• fix(traffic_light_classifier): add arg data_path, update model path #5144
• fix(tensorrt yolo): update model path #5158
• fix(lidar_centerpoint): add arg data_path, update model path #5162
• fix(image_projection_based_fusion): add arg data_path, update model_path #5163

Purpose

Be able to build the packages without requiring to download the ONNX models

Possible approaches

Remove the download logic so that ONNX models, update ansible scripts to download them when deploying the packages and document how to manually download the models.

Definition of done

Packages can be built without downloading the ONNX models. Documentation is updated to let users manually download the ONNX models. Update ansible scripts to download the models.

[yukkysaito]

Thank you for your proposal.
If DOWNLOAD_ARTIFACTS is default false, won't the rosbag replay simulator stop working?

[esteve]

@yukkysaito thanks for your comment. I've only submitted changes to the packages that download ONNX models. Could you link to the packages that download rosbag files? Thanks.

In any case, the reason for not downloading external files by default is because the processes we use for generating Debian packages restrict access to the network (as is the Debian policy for building packages), so we can't download files in CMakeLists.txt. I have some questions:

• Is there another way to distribute the rosbag files?
• Perhaps a separate package or adding instructions for users?
• How big are the files?
• Maybe we could use something like git-lfs to store them in Git?

Thanks.

[yukkysaito]

@esteve sorry, I may have written it badly and caused a misunderstanding.
I am not talking about rosbag, but rosbag replay simulator. 🙏
https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/

rosbag replay simulator uses DNN such as centerpoint, so it is necessary to download the onnx file.
If the default is set to false, the onnx file will not be downloaded and the rosbag replay simulator in the tutorial will not work.

I would preferably download it when setting up the environment in ansible, so that no internet connection is needed at build time.

[esteve]

@yukkysaito ah, sorry, I misunderstood. Thanks, I agree it'd be much easier if downloading the ONNX models is done outside the build process of CMakeLists.txt. I can remove the logic from CMakeLists.txt and we can update the ansible scripts. I'm not familiar with ansible, but I can try.

[xmfcx]

I agree, the links are in the CMakeLists.txt file and most are in https://web.auto domain name right now.

Moving them to ansible and also adding the manual steps in the readme of the role should be enough.

And yes like @esteve suggested we should remove the remains of the downloads from the CMakeLists.txt files.

So we should remove the DOWNLOAD_ARTIFACTS in general. (But if you want to add the logic and set the default to false before removing it, it's fine too)

[esteve]

Following this discussion I've closed autowarefoundation/autoware_common#164 and will rework the rest of the PRs to remove the download logic. I'll also update the documentation and the ansible scripts.

[yukkysaito]

There are a few things to consider for downloading with ansible.

• Where to put the downloads?
• How do DNN pkgs (centerpoint, etc.) refer to the downloaded file from launch?

As an example,to refer to the downloaded file in launch file by ansible, we can create a specific package, put the weight file there, and refer to it from launch.
Do you have any other suggestions?

[xmfcx]

@yukkysaito

Right now Autoware has map_path as an external directory necessary for the runtime:

ros2 launch autoware_launch logging_simulator.launch.xml \
map_path:=$HOME/autoware_map/sample-map-rosbag \
vehicle_model:=sample_vehicle \
sensor_model:=sample_sensor_kit

What about adding a ml_models_path (machine learning) path too?

Then this path could be passed to the necessary nodes.

[esteve]

There are a few things to consider for downloading with ansible.

* Where to put the downloads?

* How do DNN pkgs (centerpoint, etc.) refer to the downloaded file from launch?

As an example,to refer to the downloaded file in launch file by ansible, we can create a specific package, put the weight file there, and refer to it from launch. Do you have any other suggestions?

Sorry, I'm afraid I'm not following here. Will the package store the weight file? And if not using ansible, how will the package get populated with the model files? Could you explain further? Thanks.

[esteve]

@xmfcx

What about adding a ml_models_path (machine learning) path too?

Then this path could be passed to the necessary nodes.

I think this is a good idea, it'll standardize how to access the models across all nodes. What path by default should we tell users to download files to? With ansible we can decide for them, but I'm thinking of when Autoware is installed manually.

Also, when/if we generate Debian packages for these packages, should the models be installed system-wide or in a user's folder?

[yukkysaito]

What about adding a ml_models_path (machine learning) path too?
Then this path could be passed to the necessary nodes.

@xmfcx @mitsudome-r
I think the map path is necessary because it specifies the environment in which to run, but ml_model_path seems to be a parameter of convenience inside Autoware.
So I would like to make the user don't care it, is there a good way to do this?

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[esteve]

Closing this ticket now that all the linked PRs are merged. Thanks everyone, especially @lexavtanke