Subaru FPv2 support

[martinl]

Opening this for discussion to add Subaru FPv2 support. Also included some collected FW version identifiers for Ascent and Impreza (some Ascent trims FPv1 fingerprint as Impreza and need FPv2 to be upstreamed).

According to service manual, Subaru ECU-s have following identifiers:

• SSMID Identification Option Vehicle Manufacturer Specific Data Identifier
• ROMID application Data Identification Data Identifier

We are using ROMID (22 F1 82) to query the application identifier for each essential ECU. FPv2 matching is working for global and preglobal platform models currently in subaru-PR-test branch.

One possible downside is 10 second ECU init delay after ignition. We could not get reliable responses from all ECUs with shorter delay. Another minor known issue is that Hyundai has fwdRadar (0x7d0) and fwdCamera (0x7c4) on same addresses as some Subaru non-essential ECUs. So they are also listed in carFw values in useradmin logs but can be ignored when adding values to FW_VERSIONS list.

[pd0wm]

Are you 100% sure on this 10 second delay on startup? That would be a dealbreaker. That adds a super long delay to startup, and would basically always fault Toyotas with DSU unplugged.

[martinl]

I did test different combinations of UDS tester present and diagnostic request commands with Subaru version request. I also tried to reduce the delay to 5 seconds, which gave same result as no delay (at least one essential ecu was missing from responses or had negative response and FPv2 did not match). I'm hoping there is a way to speed it up and I'm open to suggestions

[pd0wm]

What do you get as a response when you try to go into extended diagnostics mode immediately on startup? What are the specific error codes you're getting?

[martinl]

Here are some examples from 2018 Subaru Forester 2.5i Touring

only SUBARU_VERSION_REQUEST (22 F1 82), no delay:

first scan:

Found FW versions
{
  (Ecu.engine, 0x7e0, None): [b'\xba"@p\x07']
  (Ecu.esp, 0x7b0, None): [b'}\x97\x14@']
  (Ecu.unknown, 0x742, None): [b'\xa1\x00\x00']
  (Ecu.unknown, 0x743, None): [b'\xa1\x10\x00']
  (Ecu.unknown, 0x746, None): [b'}\xc0\x10\x00']
  (Ecu.unknown, 0x747, None): [b'\x172$\x00']
  (Ecu.unknown, 0x752, None): [b"'\x81\x9c\x06"]
  (Ecu.unknown, 0x751, None): [b'\x12']
  (Ecu.unknown, 0x776, None): [b'\x00\x00\x16\x00\x00']
  (Ecu.unknown, 0x783, None): [b'\x00\x00\x01']
  (Ecu.unknown, 0x786, None): [b'\xab\x0b\x81']
  (Ecu.unknown, 0x787, None): [b'\x00\x00d5\x00\x00\x00\x00']
  (Ecu.unknown, 0x7d5, None): [b'\xf1\x00\x00']
  (Ecu.unknown, 0x7c4, None): [b'!\x03\x10\x00\x00']
  (Ecu.unknown, 0x7f1, None): [b'\x90\x00%\x00']
}

Possible matches: set()

second scan:

Found FW versions
{
  (Ecu.engine, 0x7e0, None): [b'\xba"@p\x07']
  (Ecu.esp, 0x7b0, None): [b'}\x97\x14@']
  (Ecu.unknown, 0x742, None): [b'\xa1\x00\x00']
  (Ecu.unknown, 0x746, None): [b'}\xc0\x10\x00']
  (Ecu.unknown, 0x743, None): [b'\xa1\x10\x00']
  (Ecu.unknown, 0x747, None): [b'\x172$\x00']
  (Ecu.unknown, 0x752, None): [b"'\x81\x9c\x06"]
  (Ecu.unknown, 0x751, None): [b'\x12']
  (Ecu.unknown, 0x776, None): [b'\x00\x00\x16\x00\x00']
  (Ecu.unknown, 0x783, None): [b'\x00\x00\x01']
  (Ecu.unknown, 0x786, None): [b'\xab\x0b\x81']
  (Ecu.unknown, 0x780, None): [b'\x90\x00%\x00']
  (Ecu.unknown, 0x787, None): [b'\x00\x00d5\x1f@ \t']
  (Ecu.unknown, 0x7c4, None): [b'!\x03\x10\x00\x00']
  (Ecu.unknown, 0x7d5, None): [b'\xf1\x00\x00']
  (Ecu.unknown, 0x7f1, None): [b'\x90\x00%\x00']
}

0x780 shows up in second scan, no reply from transmission (0x7e1)

running ecu_list_subaru.py (22 F1 97), no delay:

first scan

Found ECU descriptions
{
  (Ecu.unknown, 0x7e0, None): [b'2.5 DOHC                        ']
  (Ecu.unknown, 0x7b0, None): [b'VDC                             ']
  (Ecu.unknown, 0x7e1, None): [b'\xf1\x00\xa4\x10@']
  (Ecu.unknown, 0x787, None): [b'EyeSight System                 ']
  (Ecu.unknown, 0x742, None): [b'\xf1\x00\xa1\x00\x00']
  (Ecu.unknown, 0x743, None): [b'RADAR ASSY B&S RH               ']
  (Ecu.unknown, 0x752, None): [b'Integ. Unit mode                ']
  (Ecu.unknown, 0x747, None): [b'Automatic Headlight Leveling    ']
  (Ecu.unknown, 0x751, None): [b'Keyless Access System           ']
  (Ecu.unknown, 0x757, None): [b'\xf1\x00\x01\x01\x01']
  (Ecu.unknown, 0x746, None): [b'Power Steering System           ']
  (Ecu.unknown, 0x776, None): [b'Data Communication Module\x00\x00\x00\x00\x00\x00\x00']
  (Ecu.unknown, 0x783, None): [b'METER                           ']
  (Ecu.unknown, 0x786, None): [b'Sonar system                    ']
  (Ecu.unknown, 0x780, None): [b'\xf1\x00X\x06\x00']
  (Ecu.unknown, 0x7c4, None): [b'\xf1\x00\xac\x00\x00']
  (Ecu.unknown, 0x7d5, None): [b'Multi Function Display          ']
  (Ecu.unknown, 0x7f1, None): [b'\xf1\x00X\x06\x00']
}

second scan

Found ECU descriptions
{
  (Ecu.unknown, 0x7c4, None): [b'\xf1\x00\xac\x00\x00']
  (Ecu.unknown, 0x7e1, None): [b'\xf1\x00\xa4\x10@']
  (Ecu.unknown, 0x7e0, None): [b'\xf1\x00\xa2\x10\n']
  (Ecu.unknown, 0x780, None): [b'Airbag System\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f']
  (Ecu.unknown, 0x742, None): [b'RADAR ASSY B&S LH               ']
  (Ecu.unknown, 0x743, None): [b'\xf1\x00\xa1\x10\x00']
  (Ecu.unknown, 0x751, None): [b'Keyless Access System           ']
  (Ecu.unknown, 0x753, None): [b'\xf1\x00\xca\x01\x00']
  (Ecu.unknown, 0x752, None): [b'Integ. Unit mode                ']
  (Ecu.unknown, 0x747, None): [b'Automatic Headlight Leveling    ']
  (Ecu.unknown, 0x776, None): [b'Data Communication Module\x00\x00\x00\x00\x00\x00\x00']
  (Ecu.unknown, 0x786, None): [b'\xf1\x00\xab\x0b\x81']
  (Ecu.unknown, 0x7b0, None): [b'VDC                             ']
  (Ecu.unknown, 0x787, None): [b'\xf1\x00\xf0\xe0\x0e']
  (Ecu.unknown, 0x783, None): [b'METER                           ']
  (Ecu.unknown, 0x7d5, None): [b'\xf1\x00\xf1\x00\x00']
  (Ecu.unknown, 0x7f1, None): [b'Airbag System\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f\x7f']
}

• After adding TESTER_PRESENT_REQUEST we got replies from all ecus when car is running but still missed some responses when fingerprinting after ignition
• After adding 10 second delay before fingerprinting we got reliably all responses from all essential ecus during fingerprinting v2

I'll test with extended diagnostics again, thanks for the suggestions

[pd0wm]

Can you send a drive where it doesn't fingerprint? Like to check out the raw response. The fingerprinting library strips off some of the response.

Also b'Power Steering System ' is not a very useful identifier since it contains no version or car specific information. Are there other identifiers that return something useful (https://github.com/commaai/panda/blob/master/python/uds.py#L99)?

[martinl]

Can you send a drive where it doesn't fingerprint? Like to check out the raw response. The fingerprinting library strips off some of the response.

Also b'Power Steering System ' is not a very useful identifier since it contains no version or car specific information. Are there other identifiers that return something useful (https://github.com/commaai/panda/blob/master/python/uds.py#L99)?

we use SUBARU_VERSION_REQUEST (22 F1 82) responses from first two scans for FPv2 FW version values as seen in the PR change list.

ECU description query (22 F1 97) is just another example of similar behaviour (missing and negative responses). ECU description strings are mostly useful for identifying which type of ECU is on which address.

I'll get some drives by tomorrow

[martinl]

I added CAR.IMPREZA to IGNORED_FINGERPRINTS and did some FPv2 only tests

Here are some recorded logs:

with 10 second delay in car_helpers.py (fingerprinted correctly as Impreza 2019):
05bca04dfbdca165|2020-07-17--11-00-17

without 10 second delay (mock):
05bca04dfbdca165|2020-07-17--10-56-05
05bca04dfbdca165|2020-07-17--10-58-02

I also tested following diagnostic request combinations without a delay, all resulted mock detection:

• TESTER_PRESENT_REQUEST, DEFAULT_DIAGNOSTIC_REQUEST, EXTENDED_DIAGNOSTIC_REQUEST, SUBARU_VERSION_REQUEST
• DEFAULT_DIAGNOSTIC_REQUEST, EXTENDED_DIAGNOSTIC_REQUEST, SUBARU_VERSION_REQUEST
• DEFAULT_DIAGNOSTIC_REQUEST, SUBARU_VERSION_REQUEST
• EXTENDED_DIAGNOSTIC_REQUEST, SUBARU_VERSION_REQUEST

Here are some of the drives with diagnostic requests:
05bca04dfbdca165|2020-07-17--11-02-24
05bca04dfbdca165|2020-07-17--11-05-08
05bca04dfbdca165|2020-07-17--11-08-26

[martinl]

I also scanned the OBD Mode 22 ranges quite extensively but did not find anything more usable for FPv2 than with first tests here #1238 (comment)

[pd0wm]

I'll look at the drives on Monday.

Have you tried logging the manufacturer software? They might be using some undocumented stuff like Toyota and Hyundai.

[martinl]

I'm looking at options to getting access to manufacturer software, might take a few weeks since most people are currently on vacation

[pd0wm]

I looked at "05bca04dfbdca165|2020-07-17--10-56-05", and the messages itself look completely fine. Positive responses, and nothing weird going on. But you're saying the data that it returns is not consistent?

I think we really need to have a look what the manufacturer software does.

bus, addr, msg
129 0x746 b'0322f18200000000'
129 0x7e0 b'0322f18200000000'
129 0x7e1 b'0322f18200000000'
1 0x7e8 b'100862f182aa6166'
1 0x74e b'0762f1827ac00c00'
1 0x7e9 b'100862f182e3e546'
1 0x78f b'027e000000000000'
1 0x7b8 b'027e000000000000'
129 0x7e0 b'3000000000000000'
129 0x7e1 b'3000000000000000'
1 0x7e9 b'2131000000000000'
1 0x7e8 b'2173070000000000'
129 0x7b0 b'0322f18200000000'
129 0x787 b'0322f18200000000'
1 0x78f b'100b62f182000064'
129 0x787 b'3000000000000000'
1 0x7b8 b'100862f1827a943f'
129 0x7b0 b'3000000000000000'
1 0x78f b'21b5000000000000'
1 0x7b8 b'2190000000000000'

[martinl]

I extracted carFw values from last 3 routes, the results describe inconsistent and missing values issue quite well

route id	engine (0x7e0)	fwdCamera (0x787)	transmission (0x7e1)	esp (0x7b0)	eps (0x746)
05bca04dfbdca165|2020-07-17--11-08-26	\xf1\x00\xa2\x10\t		\xf1\x00\xa4\x10@	z\x94?\x90\x00
05bca04dfbdca165|2020-07-17--11-05-08	\xf1\x00\xa2\x10\t		\xf1\x00\xa4\x10@	z\x94?\x90\x00
05bca04dfbdca165|2020-07-17--11-02-24--0	\xaaafs\a	\x00\x00d\xb5\x00\x00\x00\x00	\xe3\xe5F1\x00	z\x94?\x90\x00	z\xc0\x0c\x00
Crosstrek 2018 FPv2	\xaaafs\a	\x00\x00d\xb5\x1f@ \x0e	\xe3\xe5F1\x00	z\x94?\x90\x00	z\xc0\x0c\x00

I'll see if I can find out how the official software does ecu firmware version identification

[martinl]

I asked around and currently first chance of getting access to vendor software is in end of Aug - beginning of Sept timeframe

[martinl]

Scheduled SSM4 access for next Friday

[adeebshihadeh]

Any update on this?

[martinl]

so far no progress, I'll try to reschedule next week

[zchstdt]

Hey all, I was just curious in regards to the 10s delay thing. Is the main issue with the 10s delay that it will mess with Toyota stuff? If it is, is there any reason we couldn't use the VIN that we get a couple of lines earlier to determine that the vehicle is a subaru and then wrap the time.sleep in an if statement so that it only happens if it's a subaru VIN thereby not messing with Toyota?

[martinl]

I think checking VIN would be worth a try until we have a better solution. VIN-s starting with JF1, JF2, 4S3 and 4S4 should cover most Subarus since 2015 for AU/EU/US markets.

As a side note, VIN is not used and required in Japan, so it would be worth checking if Subaru JDM cars have it or not

[zchstdt]

Is there anything special we have to do to get Comma staff to weigh in on whether or not this would work for upstreaming Subarus?

[pd0wm]

Has been discussed quite a bit, and unfortunately VINs are not usable for fingerprinting. When I looked at the data there are extra WMI that show up that are not documented. Also japan doesn't use VIN and just returns a frame number when you request a VIN.

[COValhalla]

If I understand this correctly we aren't using VINs for fingerprinting. We are simply saying if you detect these VINs wait 10 seconds then fingerprint as normal (FPv2). This pulls in the majority (if not all) of the Preglobal Platform, which is the older cars. Global (newer cars) are already upstreamed.

[zchstdt]

Another thought, rather than using a VIN to detect subaru, could we just fingerprint with the normal 1 second delay and then if it determines the platform is "None" or "mock". It waits 10 seconds and tries again? Are there other ramifications to that?

[martinl]

I gated the 10 second delay behind VIN code Subaru WMI match check. It is currently implemented for testing and feedback in my subaru-PR-test branch martinl@8ab535d

Subaru JDM models should have 7-character applied model code instead of VIN: http://www.subarujdm.com/jdm_impreza_vin_models_wrx_sti_gc8_gf8.html

It does not look promising since applied model code does not include unique manufacturer identifier

[gregjhogan]

can you set the debug=True param here and provide the full output when an ECU doesn't respond?

openpilot/selfdrive/car/fw_versions.py

Line 173 in 204d878

def get_fw_versions(logcan, sendcan, bus, extra=None, timeout=0.1, debug=False, progress=False):

it is possible that a response pending status is getting returned and we don't currently wait for that to resolve
https://github.com/commaai/panda/blob/d6d928125b7f26003aae64be9bb1a89fe76c6b8f/python/uds.py#L249

[martinl]

I ran ./fw_versions.py --debug twice, once immediately after ignition (FPv2 fail) and second time about 10 seconds later (FPv2 success). I did not get any missing ecus but Eyesight (0x787) responded with different ROM ID than 10 seconds later. I can ask some preglobal model owners to run the same, missing ecu responses were quite consistent there.

Both outputs are attached here:
subaru-fw-query-fail-2021-04-04.txt
subaru-fw-query-success-2021-04-04.txt

[gregjhogan]

Yeah we will need the debug log from when an ecu is missing to see if that problem is the code not handling the response pending status

[martinl]

You are correct, there is no 1 second delay when get_fw_versions is called from car_helpers. I think I mixed it with 1 second delay before vin query when running fw_versions.py directly from command line.

[adeebshihadeh]

How's the test going on your fork? Are you still having issues with no delay?

[martinl]

I have received feedback from about 10 users with different models who are running subaru-PR-test or subaru-community without the delay and we have not been able to reproduce the missing ecu responses issue during last month or so. I think this is good to merge.

[pd0wm]

This query is a subset of the VW query, I therefore suggest we put it before the VW one. That would change the timing, so might require some more testing on the Subaru side. Just rebased this branch on master and pushed that change.

There is an overlap in addresses with the following brands that need to be tested before merging. This query might elicit a response from ECUs on other brands and cause issues. However, since the query is now before VW, Nissan and Mazda I don't expect issues with those brands.

• Subaru (after change in order)
• Toyota (ESP 0x7b0, Engine 0x7e0)
• VW (Engine 0x7e0, Transmission, 0x7e1)
• Nissan (Engine 0x7e0)
• Mazda (Engine 0x7e0, Transmission 0x7e1)
• Hyundai (Engine 0x7e0, Transmission 0x7e1)

@adeebshihadeh Can you test on Toyota and Hyundai?

For the other brands, the following people have worked on initially upstreaming FW fingerprinting for their brands. It would be very appreciated if you can find the time to quickly test this PR and post a route with rlogs.
@robin-reckmann for Nissan, @Jafaral for Mazda and @jyoung8607 for VW.

[jyoung8607]

@jyoung8607 for VW.

LGTM: 3cfdec54aa035f3f|2021-12-15--11-15-06

[robin-reckmann]

Nissan Leaf: no issues ;)

fd3b0ef05a984a10|2021-12-16--17-16-13

[robin-reckmann]

Oh wait, the leaf doesn't have the engine ecu on 7e0 . Oops

@avolmensky: could you test this with the x-trail?

[adeebshihadeh]

Corolla: 7f14d4258db3be75|2021-12-16--14-23-47
Sonata: 73d9fe6ccb8b4266|2021-12-16--19-01-17

[Jafaral]

Mazda 8c521516df3527d8|2021-12-16--20-32-15

[avolmensky]

Nissan X-Trail: 2c976ff547f21516|2021-12-17--07-32-27

[pd0wm]

The sonata transmission unfortunately returns a response to this query as well. So I'm going to have to move Subaru all the way to the top of the list.

@martinl let me know if you see any timing issues due to this.
@adeebshihadeh Can you test again on the Sonata?

[martinl]

I did a few tests and FPv2 worked, one of the routes from today is 05bca04dfbdca165|2021-12-18--12-40-44

[adeebshihadeh]

New Sonata route: 73d9fe6ccb8b4266|2022-01-03--18-13-50

[pd0wm]

Looks good!