Reth panicked with "internal error: entered unreachable code: Account ..."

[tobernguyen]

Describe the bug

Reth crashed. These are the logs before it exited.

2023-12-21T05:36:46.840951Z  INFO reth::cli: Status connected_peers=71 freelist=2837806 latest_block=18832127
2023-12-21T05:36:55.561386Z  INFO reth::node::events: Canonical chain committed number=18832139 hash=0xa2eddfc0b1255c6c19cf4b7a591f17f21f7f89073a8d8ca127981d1d0f230902 elapsed=18.770867421s
2023-12-21T05:36:55.572067Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xcefd9dc23f8693ee9ebd87862ecb0f8e0560db1aa369111beb17fec1fcdb4800 safe_block_hash=0x57939d2790abd333486d43b8a4686ec996aca7587a6995c2f4508b27297cdb50 finalized_block_hash=0xecc6619e5be8c6666651d1d5f4eeec4b9f91e796d4a2ea9e094f774e5a5e760f status=Syncing
2023-12-21T05:36:55.578026Z  INFO reth::node::events: Forkchoice updated head_block_hash=0xa056b73331216efc5f0c111f47c2d2174b6099d22f723c040cbfa047a191a8f6 safe_block_hash=0xa2eddfc0b1255c6c19cf4b7a591f17f21f7f89073a8d8ca127981d1d0f230902 finalized_block_hash=0x57939d2790abd333486d43b8a4686ec996aca7587a6995c2f4508b27297cdb50 status=Syncing
2023-12-21T05:36:55.578200Z  WARN consensus::engine: Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator. hook=Prune
2023-12-21T05:36:55.578248Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x67d14887ea672fa6d4a197c42df49e2439383f66c31e0b1ed399c1ea318ac942 safe_block_hash=0xa2eddfc0b1255c6c19cf4b7a591f17f21f7f89073a8d8ca127981d1d0f230902 finalized_block_hash=0x57939d2790abd333486d43b8a4686ec996aca7587a6995c2f4508b27297cdb50 status=Syncing
2023-12-21T05:36:55.652115Z  INFO reth::node::events: Pruner finished tip_block_number=18832139 elapsed=73.045531ms stats={SenderRecovery: (HasMoreData, 42000), Receipts: (Finished, 0)}
2023-12-21T05:36:55.677051Z  INFO reth::node::events: Forkchoice updated head_block_hash=0x67d14887ea672fa6d4a197c42df49e2439383f66c31e0b1ed399c1ea318ac942 safe_block_hash=0xa2eddfc0b1255c6c19cf4b7a591f17f21f7f89073a8d8ca127981d1d0f230902 finalized_block_hash=0x57939d2790abd333486d43b8a4686ec996aca7587a6995c2f4508b27297cdb50 status=Syncing
thread 'tokio-runtime-worker' panicked at /cargo/git/checkouts/revm-bf744d8ffabcbad9/b00ebab/crates/revm/src/db/states/bundle_state.rs:636:21:
internal error: entered unreachable code: Account 0x00000000009e50a7ddb7a7b0e2ee6604fd120e49 AccountRevert { account: RevertTo(AccountInfo { balance: 0x0000000000000000000000000000000000000000000000000000037f5ab5fcf1_U256, nonce: 1, code_hash: 0xdce768b5fe98e230466535dcd97f7b35e9c2958fd8ad4c339ddd2ed11eddc6dc, code: None }), storage: {}, previous_status: Loaded, wipe_storage: true } for revert should exist
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-12-21T05:37:00.059742Z ERROR reth_tasks: Critical task `consensus engine` panicked: `internal error: entered unreachable code: Account 0x00000000009e50a7ddb7a7b0e2ee6604fd120e49 AccountRevert { account: RevertTo(AccountInfo { balance: 0x0000000000000000000000000000000000000000000000000000037f5ab5fcf1_U256, nonce: 1, code_hash: 0xdce768b5fe98e230466535dcd97f7b35e9c2958fd8ad4c339ddd2ed11eddc6dc, code: None }), storage: {}, previous_status: Loaded, wipe_storage: true } for revert should exist`
2023-12-21T05:37:00.077203Z  INFO reth::cli: Wrote network peers to file peers_file="/reth/known-peers.json"
Error: Critical task `consensus engine` panicked: `internal error: entered unreachable code: Account 0x00000000009e50a7ddb7a7b0e2ee6604fd120e49 AccountRevert { account: RevertTo(AccountInfo { balance: 0x0000000000000000000000000000000000000000000000000000037f5ab5fcf1_U256, nonce: 1, code_hash: 0xdce768b5fe98e230466535dcd97f7b35e9c2958fd8ad4c339ddd2ed11eddc6dc, code: None }), storage: {}, previous_status: Loaded, wipe_storage: true } for revert should exist`
Location:
    /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/convert/mod.rs:716:9

Steps to reproduce

I don't know how to reproduce it.

Node logs

reth-2023-12-22.log

Platform(s)

Linux (x86)

What version/commit are you on?

0.1.0-alpha.13

What database version are you on?

Current database version: 1

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[tobernguyen]

This occurred again yesterday.

2023-12-25T08:30:20.800286Z  INFO reth::cli: Status connected_peers=37 freelist=2020266 latest_block=18861498
thread 'tokio-runtime-worker' panicked at /cargo/git/checkouts/revm-bf744d8ffabcbad9/b00ebab/crates/revm/src/db/states/bundle_state.rs:636:21:
internal error: entered unreachable code: Account 0x660000c7feae0027f565680000005c683400e700 AccountRevert { account: RevertTo(AccountInfo { balance: 0x000000000000000000000000000000000000000000000000000000013d6950a9_U256, non
ce: 1, code_hash: 0xa10c2b5ce1690e99aa3cbbc49e3e984293c9e4cb9106a24f0488ed22d8d977f5, code: None }), storage: {}, previous_status: Loaded, wipe_storage: true } for revert should exist
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-12-25T08:30:25.136204Z ERROR reth_tasks: Critical task `consensus engine` panicked: `internal error: entered unreachable code: Account 0x660000c7feae0027f565680000005c683400e700 AccountRevert { account: RevertTo(AccountIn
fo { balance: 0x000000000000000000000000000000000000000000000000000000013d6950a9_U256, nonce: 1, code_hash: 0xa10c2b5ce1690e99aa3cbbc49e3e984293c9e4cb9106a24f0488ed22d8d977f5, code: None }), storage: {}, previous_status: Loade
d, wipe_storage: true } for revert should exist`
2023-12-25T08:30:25.145283Z  INFO reth::cli: Wrote network peers to file peers_file="/reth/known-peers.json"
Error: Critical task `consensus engine` panicked: `internal error: entered unreachable code: Account 0x660000c7feae0027f565680000005c683400e700 AccountRevert { account: RevertTo(AccountInfo { balance: 0x00000000000000000000000
0000000000000000000000000000000013d6950a9_U256, nonce: 1, code_hash: 0xa10c2b5ce1690e99aa3cbbc49e3e984293c9e4cb9106a24f0488ed22d8d977f5, code: None }), storage: {}, previous_status: Loaded, wipe_storage: true } for revert shou
ld exist`
Location:
    /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/convert/mod.rs:716:9

This is the debug log.

reth-2023-12-25.log.zip

[rakita]

This is interesting one, problem is the account is missing from bundle that we want to revert.
https://github.com/bluealloy/revm/blob/75e95675e32e1f16923502829720a174448d5569/crates/revm/src/db/states/bundle_state.rs#L640

But when new transitions are applied we don't remove anything from the state: https://github.com/bluealloy/revm/blob/75e95675e32e1f16923502829720a174448d5569/crates/revm/src/db/states/bundle_state.rs#L417-L436

But we are removing the account when the Revert account is AccountInfoRevert::DeleteIt: https://github.com/bluealloy/revm/blob/75e95675e32e1f16923502829720a174448d5569/crates/revm/src/db/states/bundle_state.rs#L635C28-L635C28
True here:
https://github.com/bluealloy/revm/blob/75e95675e32e1f16923502829720a174448d5569/crates/revm/src/db/states/bundle_account.rs#L96

Will need to think about the fix, probably just using EmptyAccount there.

[joshieDo]

fixed by the above revm pr