Failed to register the Ethereum service: gap (#15565418) in the chain between ancients and leveldb

[danspina1]

System information

Geth version: 1.10.25
CL client & version: prysm@v3.1.1
OS & Version: Ubuntu 20.04
Running inside a self built container
Commit hash : (if develop)

Background info

Node is an archive node, I used a snapshot from another server to build the new server

Actual behaviour

{"lvl":"info","msg":"Enabling metrics collection","t":"2022-10-06T07:19:31.836176092Z"}
{"ETH":50,"LES":0,"lvl":"info","msg":"Maximum peer count","t":"2022-10-06T07:19:31.839022609Z","total":50}
{"err":"stat /run/pcscd/pcscd.comm: no such file or directory","lvl":"info","msg":"Smartcard socket not found, disabling","t":"2022-10-06T07:19:31.840177696Z"}
{"lvl":"info","msg":"Enabling recording of key preimages since archive mode is used","t":"2022-10-06T07:19:31.844613528Z"}
{"cap":50000000,"lvl":"info","msg":"Set global gas cap","t":"2022-10-06T07:19:31.8449131Z"}
{"clean":"920.00 MiB","dirty":"0.00 B","lvl":"info","msg":"Allocated trie memory caches","t":"2022-10-06T07:19:31.846210128Z"}
{"cache":"1.50 GiB","database":"/data/geth/chaindata","handles":524288,"lvl":"info","msg":"Allocated cache and file handles","t":"2022-10-06T07:19:31.846270784Z"}
{"location":"/data/geth/chaindata/ancient","lvl":"info","msg":"Found legacy ancient chain path","t":"2022-10-06T07:20:07.662925882Z"}
{"database":"/data/geth/chaindata/ancient","lvl":"info","msg":"Opened ancient database","readonly":"false","t":"2022-10-06T07:20:07.72458557Z"}
Fatal: Failed to register the Ethereum service: gap (#15565418) in the chain between ancients and leveldb
Fatal: Failed to register the Ethereum service: gap (#15565418) in the chain between ancients and leveldb
{"addr":"http://0.0.0.0:6060/debug/pprof","lvl":"info","msg":"Starting pprof server","t":"2022-10-06T07:20:08.398647Z"}
{"lvl":"info","msg":"Starting Geth on Ethereum mainnet...","t":"2022-10-06T07:20:08.399091259Z"}
{"lvl":"info","msg":"Enabling metrics collection","t":"2022-10-06T07:20:08.39911289Z"}
{"ETH":50,"LES":0,"lvl":"info","msg":"Maximum peer count","t":"2022-10-06T07:20:08.400893481Z","total":50}
{"err":"stat /run/pcscd/pcscd.comm: no such file or directory","lvl":"info","msg":"Smartcard socket not found, disabling","t":"2022-10-06T07:20:08.401551458Z"}
{"lvl":"info","msg":"Enabling recording of key preimages since archive mode is used","t":"2022-10-06T07:20:08.403818968Z"}
{"cap":50000000,"lvl":"info","msg":"Set global gas cap","t":"2022-10-06T07:20:08.405123109Z"}
{"clean":"920.00 MiB","dirty":"0.00 B","lvl":"info","msg":"Allocated trie memory caches","t":"2022-10-06T07:20:08.407748814Z"}
{"cache":"1.50 GiB","database":"/data/geth/chaindata","handles":524288,"lvl":"info","msg":"Allocated cache and file handles","t":"2022-10-06T07:20:08.407839918Z"}
{"location":"/data/geth/chaindata/ancient","lvl":"info","msg":"Found legacy ancient chain path","t":"2022-10-06T07:20:44.0721435Z"}
{"database":"/data/geth/chaindata/ancient","lvl":"info","msg":"Opened ancient database","readonly":"false","t":"2022-10-06T07:20:44.121151479Z"}
Fatal: Failed to register the Ethereum service: gap (#15565418) in the chain between ancients and leveldb
Fatal: Failed to register the Ethereum service: gap (#15565418) in the chain between ancients and leveldb

Steps to reproduce the behaviour

Backtrace

[backtrace]

When submitting logs: please submit them as text and not screenshots.

[holiman]

Could you try

 geth  --datadir <your datadir>  db get LastHeader

Should produce something like

key 0x4c617374486561646572: 0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3

The value there (0xd4e5... in my example) is a header hash. Now take that, whatever value you have, prefix it with a H (0x48), and do

geth --datadir <your datadir> db get 0x48d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3

Which for me (with an empty database) gives

key 0x48d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3: 0x0000000000000000

This should tell us the value of this check *ReadHeaderNumber(db, ReadHeadHeaderHash(db)) that cause the error above to be raised:

if *ReadHeaderNumber(db, ReadHeadHeaderHash(db)) > frozen-1

[danspina1]

Commands above returned an error ailed to register the Ethereum service: gap (#15565418) in the chain between ancients and leveldb