Design of Autoware + AWSIM evaluation procedure

[WJaworskiRobotec]

Description

To create a design of tests for evaluation of Autoware stack with AWSIM. The validation process should be defined in more structured way, based on the previously performed tests:

• AWSIM feedback #67
• https://github.com/orgs/autowarefoundation/discussions/3813
• https://github.com/orgs/autowarefoundation/discussions/3741#discussioncomment-6779568

Purpose

• To make AWSIM widely used in development of Autoware

Definition of Done

• Tests scenario description prepared (OpenScenario scenarios - to get the instruction from Piotr Zyskowski, to add the instruction to the autoware-documentation website)
• Definition of success / failure (to deeply analyze failed scenarios)
• Template of test results prepared (output from each experiments set)
• Suggestion of how to handle encountered bugs (how to split them between AWSIM and Autoware etc.)

[lchojnack]

I've prepared a wiki page where I've started to describe how Autoware AWSIM verification can be done.

For test execution, I think we can use scenario_simulator_v2. As I know, the support for executing scenarios using AWSIM is in progress and will be available in the near future.

In order to think about and write down scenarios/test cases, I've prepared a spreadsheet. At the beginning we can try to reproduce the issues encountered in the previously performed tests (in the description)

Please let me know your feedback

[lchojnack]

I've started to prepare the AWSIM simulation which supports scenario_simulator_v2 scenarios.
As a first step, I've used the ITRI scenario from here

I've edited the scenario in the following way

@@ -31,6 +31,7 @@ OpenSCENARIO:
         Vehicle:
           name: ''
           vehicleCategory: car
+          model3d: "lexus_rx450h"
           BoundingBox:
             Center:
               x: 1.48
@@ -70,6 +71,7 @@ OpenSCENARIO:
         Vehicle:
           name: ''
           vehicleCategory: car
+          model3d: "taxi"
           BoundingBox:
             Center:
               x: 0

---

Then, I prepared the AWSIM with ITRI scene and ss2 support, and I ran the AWSIM with the scenario. The result is presented in the video below

itri_awsim_ss2.mp4

---

At this moment, SS2 is able to spawn EGO and NPC and start the scenario execution. There is an issue that EGO is not able to detect the NPC and crash it. For that reason, the scenario fails. I'm investigating why the issue happens.

---

Tested with

AWSIM	tier4/AWSIM@df76d6f
Unitypackage	ITRI_Campus_r02_HDRP.unitypackage
Autoware	https://github.com/autowarefoundation/autoware/tree/2023.10
SS2	tier4/scenario_simulator_v2@e187b8f

[lchojnack]

I've retested the scenario with an adjusted ITRI map to ll2. The scenario passes successfully. In the next steps, I will test the prepared AWSIM binary with other scenarios.

itri_awsim_ss2_v2.mp4

---

I've also checked if the issue mentioned here still happens and it seems that is solved, so EGO is not driving below ll2 according to Autoware

itri_awsim_issue_retest.mp4

[lchojnack]

I've prepared the AWSIM binary of ITRI scene which supports ss2 and modified openscenarios. There are available here

[lchojnack]

I've started to work on the testing scenarios. I've prepared an initial version of scenarios that verify the following planning features:

• Lane Change
• Obstacle Avoidance

So far, the scenarios are available on gdrive, but I think storing the scenarios in the GitHub repository could be a good idea.

Lane Change

scenario

PC_1.mp4

Obstacle Avoidance

scenario

PC_2.mp4

[lchojnack]

I've created one more scenario when a pedestrian walks on a crosswalk.

PedestrianOnCrosswalk

scenario

PedestrianOnCrosswalkAwsim.mp4

Conclusion

I've noticed that when the scenario is run with AWSIM, the EGO doesn't detect the NPC correctly and finally hits it.
When the scenario is run only with SS2, then the EGO stops before the pedestrian. I think the issue is related to Autoware perception pipeline.

PedestrianOnCrosswalkSs2.mp4