Novacustom capsule tests

[philipanda]

No description provided.

[philipanda]

@SergiiDmytruk

Using the capsule with wrong certificate CapsuleApp.efi -S outputs nothing no status, but only "CapsuleMax - CapsuleFFFF".
Using the capsule with wrong guid CapsuleApp.efi -S outputs Not Ready as expected.

Using the valid capsule, the update is performed properly. Immediately after the update ethernet stops working.
I've tried reflashing the firmware manually using the regions from https://docs.dasharo.com/unified/novacustom/initial-deployment/#bios-installation and the internal programmer. Additionally I've tried changing the MAC address in the FW and capsule using DCU and trying again, but it didn't help.

ubuntu@3mdeb:~$ sudo dmesg | grep e1000e
[sudo] password for ubuntu: 
[    1.485904] e1000e: Intel(R) PRO/1000 Network Driver
[    1.485909] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.487241] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.892094] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): Failed to disable ULP
[    2.578240] e1000e 0000:00:1f.6: probe with driver e1000e failed with error -2
[    8.969174]  processor_thermal_rfim btrfs cmdlinepart videobuf2_v4l2 btrtl aesni_intel intel_rapl_msr processor_thermal_rapl snd_seq_device snd_timer mei_gsc_proxy spi_nor cec btintel intel_rapl_common spd5118 crypto_simd videodev iwlwifi cryptd btbcm snd mtd processor_thermal_wt_req intel_pmc_core rc_core mei_me i2c_i801 videobuf2_common btmtk blake2b_generic processor_thermal_power_floor rapl i2c_mux processor_thermal_mbox spi_intel_pci xor nls_iso8859_1 bluetooth mc intel_cstate cfg80211 soundcore i2c_smbus spi_intel pmt_telemetry mei intel_vpu i2c_algo_bit int340x_thermal_zone igen6_edac intel_vsec joydev pmt_class intel_scu_pltdrv intel_hid raid6_pq input_leds sparse_keymap dptf_power libcrc32c mac_hid serio_raw sch_fq_codel msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 uas usb_storage hid_multitouch hid_generic 8250_dw nvme sdhci_pci crc32_pclmul thunderbolt cqhci psmouse nvme_core e1000e i2c_hid_acpi intel_lpss_pci i2c_hid xhci_pci intel_lpss sdhci intel_pmc_mux

*edit
It might have been worth mentioning, that flashing the GBE region requires passing the region to flashrom explicitly, just like the other regions. OSFV does not do that, nor is it mentioned in the documentation, but capsules do write to the GBE region. flashrom commands for laptops should include -i gbe. After reflashing with the GBE regions filled usnig DCU and the -i gbe option, the ethernet interface works back again.

[miczyg1]

If the FW (both current and capsule) was not built with these changes: Dasharo/edk2#240 then it is an expected outcome... I.e. the GBE region was probably overwritten in your case

[SergiiDmytruk]

Using the capsule with wrong certificate CapsuleApp.efi -S outputs nothing no status, but only "CapsuleMax - CapsuleFFFF".

That sounds like Dasharo/dasharo-issues#1075 which has not been addressed for NovaCustom either.

[philipanda]

After recreating the capsules with GBE region filled using DCU, the capsule update fails with valid capsule. It seems the CI creates the capsules a bit differently. It uses the keys from edk2 repo instead of generating them using the capsule.sh script in coreboot repo. It still uses the script to create and sign the capsule though.
log.zip

That sounds like Dasharo/dasharo-issues#1075 which has not been addressed for NovaCustom either.

I've been using the capsule.sh script from the coreboot repo, which generates the keys and is using them to sign the capsule. This includes the root key.

[philipanda]

Seems that the capsule indeed has to be signed with the testing keys from the EDK2 repository, not using ones generated using the capsule.sh script. The wrong_cert capsule still only outputs CapsuleMax - CapsuleFFFF, but with the valid capsule, the update is performed. The ethernet region is still being broken despite filling it up using dcuc in the FW image from which the capsule is created.
The FW used:
v54tu.zip
v54tu.cap.zip
v54tu-rc0.zip

[SergiiDmytruk]

I've been using the capsule.sh script from the coreboot repo, which generates the keys and is using them to sign the capsule. This includes the root key.

@philipanda, there is a manual step of making EDK aware of the root key (cp $dir/CapsuleRootKey.inc ${edk_workspace}/Dasharo/DasharoPayloadPkg/) which needs to be performed before building a capsule. Were you copying the newly generated key to EDK? Otherwise FmpDxe.efi is build with a test root key, but the capsule is signed with an unrelated certificate.

[philipanda]

I've been using the capsule.sh script from the coreboot repo, which generates the keys and is using them to sign the capsule. This includes the root key.

@philipanda, there is a manual step of making EDK aware of the root key (cp $dir/CapsuleRootKey.inc ${edk_workspace}/Dasharo/DasharoPayloadPkg/) which needs to be performed before building a capsule. Were you copying the newly generated key to EDK? Otherwise FmpDxe.efi is build with a test root key, but the capsule is signed with an unrelated certificate.

No, I have not been copying it. Thank you for pointing that out. CI does not generate the keys using capsule.sh but uses the testing keys from edk2 repo, which are already there.

The Ethernet region fix is already merged Dasharo/edk2#240, so the tests can be continued

[philipanda]

@mkopec
I've built coreboot from z de85fb2, rc3, I've checked that the edk2 revision in the platform config is 04af208, that's the head of current dasharo branch where @mkopec 's changes are merged. Despite that Ethernet is not working after a manual capsule update.
The firmware and capsule files I've used:
ncm_capsule_fw.zip

I'll modify both images by setting the MAC addresses using DCU and continue

[philipanda]

Setting the MAC using dcu in both the initial FW and the FW from which a capsule is generated does not help with Ethernet.

After performing a capsule update I've flashed only the gbe region using flashrom -p internal --ifd -i gbe -w $FW_FILE.
After a reboot Ethernet worked again, but the test still fails, the update is not performed. It worked fine when I did it manually once, but now it always fails with:

Capsule Status: Out of Resources

Logs:
log.zip

FW and capsule used:
ncm_cap_fw.zip

Proof of CAP250 passing:

*edit
after a couple manual retries the update was finally performed. It seems to be random

[krystian-hebel]

@philipanda did you bump CONFIG_DRIVERS_EFI_MAIN_FW_VERSION when creating new capsule?

[philipanda]

@philipanda did you bump CONFIG_DRIVERS_EFI_MAIN_FW_VERSION when creating new capsule?

@krystian-hebel Yes, I was able to perform the update manually. The photo showing the update was taken during a successful update. With the exact same capsules and the same base firmware the update often fails with status 'Out of Resources'. And the update always breaks ethernet, which is fixable by flashing the gbe region, even using the exact same firmware image out of which the capsule was created.

[mkopec]

Looks like the migration proceeds but the actual reading of the old GbE region is failing. @SergiiDmytruk any idea why the gbe region would be all 0xFF's when read through the extended BIOS window? I see you authored this commit: Dasharo/coreboot@01c93ef

Actually it looks like everything outside of the BIOS region is 0xFFs when read through the ext BIOS window

[SergiiDmytruk]

Looks like the migration proceeds but the actual reading of the old GbE region is failing. @SergiiDmytruk any idea why the gbe region would be all 0xFF's when read through the extended BIOS window? I see you authored this commit: Dasharo/coreboot@01c93ef

Actually it looks like everything outside of the BIOS region is 0xFFs when read through the ext BIOS window

I see the same values for Meteor Lake as for other SoCs:

config EXT_BIOS_WIN_BASE
	default 0xf8000000

config EXT_BIOS_WIN_SIZE
	default 0x2000000

Maybe they should be different for MTL? In particular, there is an assumption that the top of that window is used by the flash chip (see soc/intel/common/block/fast_spi/Makefile.mk), but maybe it doesn't hold for MTL and bottom half is used instead? The lines were added by Intel employees, but it doesn't mean it's free of bugs or that everything else is in place for it to work. Maybe it could help to verify that this region is not used by anything else in coreboot and that SPI flash base=0x%lx, Host base=0x%lx, Size=0x%zx\n in coreboot's log looks as expected.

Actually, I just realized that for MSI the flash map starts with:

	SI_ALL 4M {
		SI_DESC 4K
		SI_ME
	}

	UNUSED 12M

So reads might actually return all 0xFFs without breaking anything and I don't remember verifying the contents, I think update just failed without that commit.

[mkopec]

Extended decode window starts at 0xf9000000 as expected:

ubuntu@3mdeb:~$ sudo cbmem -1 | grep -i "decode window"
[INFO ]  Fixed Decode Window: SPI flash base=0x1000000, Host base=0xff000000, Size=0x1000000
[INFO ]  Extended Decode Window: SPI flash base=0x0, Host base=0xf9000000, Size=0x1000000

Reading RW_FWID_A works:

ubuntu@3mdeb:~$ sudo mmio_dump 0xf9ffffbf 0x40
0x00000000f9ffffbf: 0x746f4eff 0x6f6f6265 0x35565f6b 0x365f7834
0x00000000f9ffffcf: 0x55545f78 0x302e3176 0x722d302e 0x00003363
0x00000000f9ffffdf: 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000f9ffffef: 0x00000000 0x00000000 0x00000000 0x00000000

But reading stuff before the (IFD) BIOS region seems to fail with all FFs (this should contain the descriptor signature):

ubuntu@3mdeb:~$ sudo mmio_dump 0xf9000000 0x40
0x00000000f9000000: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
0x00000000f9000010: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
0x00000000f9000020: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
0x00000000f9000030: 0xffffffff 0xffffffff 0xffffffff 0xffffffff

To me this looks like the extended BIOS decode window can only decode the IFD BIOS region, although Intel docs aren't clear on this

[SergiiDmytruk]

To me this looks like the extended BIOS decode window can only decode the IFD BIOS region, although Intel docs aren't clear on this

Good guess. This would imply that we're not updating ME on MSI and can't change layout on capsules updates. Sounds like flash might need to be accesses via a combination of fast SPI and real SPI interfaces or, much simpler, fast SPI needs to be disabled (no idea how much faster it is).

[mkopec]

This would imply that we're not updating ME on MSI and can't change layout on capsules updates.

I think writing does work, and only reading is failing, because the GbE region after migration is full of FFs instead of whatever was there before

[krystian-hebel]

Why not use https://github.com/Dasharo/osfv-test-data/tree/master/uefi-shell?

Also, this isn't proper license for something that came from edk2 code.