beaconblockproposer: "error": "transactions root mismatch", "message":"Failed to obtain blinded proposal"

[adaszko]

Hi 👋 Any ideas what might be causing this error? It happens with Teku.

Unabridged version:
{"level":"error","service":"beaconblockproposer","impl":"standard","slot":9574572,"error":"transactions root mismatch","time":"2024-07-23T07:14:49Z","message":"Failed to obtain blinded proposal"}

This is causing VanillaBlocks production and rewards loss.

Logs around when it happens for slot 9574572: https://gist.githubusercontent.com/adaszko/d8e709c45d40c1ae1cae74352039b719/raw/7da8f53013abf929f56c7cc196ac49272f46bb6a/vouch.log

Teku produces Slot Block Production/Publishing warnings for such slots:

Jul 23 07:14:48 teku-beacon-node-1 teku-beacon-node[5466]: 07:14:48.230 WARN  - Slow Block Production *** Slot: 9574572 start 935ms, preparation_on_tick +0ms, preparation_apply_deferred_attestations +0ms, preparation_process_head +60ms, retrieve_state +0ms, beacon_block_prepared +89ms, builder_get_header +1ms, local_get_payload +8ms, beacon_block_created +1ms, state_transition +126ms, state_hashing +10ms, complete +0ms total: 295ms
Jul 23 07:14:48 teku-beacon-node-1 sh[5375]: 07:14:48.230 WARN  - Slow Block Production *** Slot: 9574572 start 935ms, preparation_on_tick +0ms, preparation_apply_deferred_attestations +0ms, preparation_process_head +60ms, retrieve_state +0ms, beacon_block_prepared +89ms, builder_get_header +1ms, local_get_payload +8ms, beacon_block_created +1ms, state_transition +126ms, state_hashing +10ms, complete +0ms total: 295ms
Jul 23 07:14:49 teku-beacon-node-1 teku-beacon-node[5466]: 07:14:49.443 WARN  - Slow Block Publishing *** Slot: 9574572 start 2181ms, blob_sidecars_prepared +16ms, block_import_completed +205ms, block_and_blob_sidecars_publishing_initiated +0ms, complete +41ms total: 262ms
Jul 23 07:14:49 teku-beacon-node-1 sh[5375]: 07:14:49.443 WARN  - Slow Block Publishing *** Slot: 9574572 start 2181ms, blob_sidecars_prepared +16ms, block_import_completed +205ms, block_and_blob_sidecars_publishing_initiated +0ms, complete +41ms total: 262ms
Jul 23 07:14:51 teku-beacon-node-1 teku-beacon-node[5466]: 07:14:51.000 INFO  - Slot Event  *** Slot: 9574572, Block: 61a70d5079a1309c05b53119738cd59a40fc3b541a1e34eafc6e2cd81f265353, Justified: 299204, Finalized: 299203, Peers: 128
Jul 23 07:14:51 teku-beacon-node-1 sh[5375]: 07:14:51.000 INFO  - Slot Event  *** Slot: 9574572, Block: 61a70d5079a1309c05b53119738cd59a40fc3b541a1e34eafc6e2cd81f265353, Justified: 299204, Finalized: 299203, Peers: 128

[adaszko]

I have a strong suspicion this is caused by

         beaconblockproposal:
            style: best
         blindedbeaconblockproposal:
            style: best

since that's one of very few changes that were made. There's 4 nodes configured in the beacon-node-addresses: section.

[mcdee]

Are your beacon node(s) configured to use the vouch instance as the builder?

[adaszko]

Yes, Ligthouse is started with the --builder <VOUCH URL> option and Teku with --builder-endpoint=<VOUCH URL>

[adaszko]

I have a strong suspicion this is caused by

         beaconblockproposal:
            style: best
         blindedbeaconblockproposal:
            style: best

since that's one of very few changes that were made. There's 4 nodes configured in the beacon-node-addresses: section.

This has been reverted and the problem is still there.

[adaszko]

Probable cause: flashbots/mev-boost#661

[adaszko]

The root cause was one of the MEV relays producing wrong payloads.