feat: a new ChainIndexer to index tipsets, messages and events

[aarshkshah1992]

For #12453.
Subsumes #12388 to implement the ChainIndexer.

TODO

• Profile Sqlite Index usage and read queries on the Index
• Clean up Config
• Smoke testing on a calibnet node (test GC, snapshot hydration, automated backfilling, etc)
• Review from @rvagg
• Write unit & itests for the ChainIndexer
• Smoke testing on @rvagg 's mainnet node.

Migration

Index "migration" will effectively involve re-indexing the chain state here. This is because the data in the old Indices isn't reliable (we're already aware of multiple bugs in the old Indices which motivated this PR in the first place) and the old Indices have never been gc'd/kept in sync with the chain state on nodes with splitstore enabled.

For Snapshot synced nodes with Splitstore enabled

• Most splitstore enabled nodes only keep 2-3 days of chain state around. The best option here is to simply backfill/re-index the chain state on node startup by setting ReConcileEmptyIndex to true and setting the MaxReconcileTipsets as needed.

For archival nodes

• If archival nodes want to backfill the index as part of daemon startup, they can always bootup the node with ReConcileEmptyIndex set to true and MaxReconcileTipsets set to infinity. This will drastically slow down the node startup but will ensure that the node starts up with it's entire history indexed.

• The other option for users is to use the Backfill RPC API with a range of height they want backfilled. However, we need to handle race conditions here where backfilling races with new tipsets coming in and need to ensure that backfilling does not end up overwriting updates caused by new tipsets being indexed.

Sqllite Query Plans

For looking up MsgInfo

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT tipset_key_cid, height FROM tipset_message WHERE message_cid='a' AN
D reverted=0"
QUERY PLAN
`--SEARCH tipset_message USING INDEX idx_message_cid (message_cid=?)

• There should be a handful of rows with reverted != 0 and only one with reverted=0 for a given message_cid so no need to index by reverted as well here.

For looking up an eth tx hash

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT message_cid FROM eth_tx_hash WHERE tx_hash = ?"
QUERY PLAN
`--SEARCH eth_tx_hash USING INDEX sqlite_autoindex_eth_tx_hash_1 (tx_hash=?)

For checking if a tipset exists

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT EXISTS(SELECT 1 FROM tipset_message WHERE tipset_key_cid = ?)"
QUERY PLAN
|--SCAN CONSTANT ROW
`--SCALAR SUBQUERY 1
   `--SEARCH tipset_message USING COVERING INDEX idx_tipset_key_cid (tipset_key_cid=?)

For getting the minimum no-reverted height for reconciliation

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT MIN(height) FROM tipset_message WHERE reverted = 0"
QUERY PLAN
`--SEARCH tipset_message USING INDEX idx_height

• Given that for a given height, there should be very few entries with reverted != 0 and only one entry with reverted=0, this should be allright.

Looking up a message_id to insert an event

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT message_id FROM tipset_message WHERE message_cid = ? AND tipset_key
_cid = ?"
QUERY PLAN
`--SEARCH tipset_message USING COVERING INDEX sqlite_autoindex_tipset_message_1 (tipset_key_cid=? AND message_cid=?)

Looking up events at a specific height

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT
   e.event_id,
   tm.height,
   hex(tm.tipset_key_cid),
   hex(e.emitter_addr),
   e.event_index,
   hex(tm.message_cid),
   tm.message_index,
   e.reverted,
   ee.flags,
   ee.key,
   ee.codec,
   ee.value
FROM event e
JOIN tipset_message tm ON e.message_id = tm.message_id
JOIN event_entry ee ON e.event_id = ee.event_id
WHERE tm.height=1954494 AND e.reverted = false
ORDER BY tm.height DESC, ee.rowid ASC"
QUERY PLAN
|--SEARCH tm USING INDEX idx_height (height=?)
|--SEARCH e USING INDEX idx_event_message_id (message_id=?)
|--SEARCH ee USING INDEX event_entry_event_id (event_id=?)
`--USE TEMP B-TREE FOR RIGHT PART OF ORDER BY

Looking up events in a height range

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT
   e.event_id,
   tm.height,
   tm.tipset_key_cid,
   e.emitter_addr,
   e.event_index,
   tm.message_cid,
   tm.message_index,
   e.reverted,
   ee.flags,
   ee.key,
   ee.codec,
   ee.value
FROM event e
JOIN tipset_message tm ON e.message_id = tm.message_id
JOIN event_entry ee ON e.event_id = ee.event_id
WHERE tm.height BETWEEN 1954481 AND 1954494
   AND e.reverted = false
ORDER BY tm.height DESC, ee.rowid ASC"
QUERY PLAN
|--SEARCH tm USING INDEX idx_height (height>? AND height<?)
|--SEARCH e USING INDEX idx_event_message_id (message_id=?)
|--SEARCH ee USING INDEX event_entry_event_id (event_id=?)
`--USE TEMP B-TREE FOR RIGHT PART OF ORDER BY

Looking up events for a specific tipset

~/.lotus-calibnet/sqlite% sqlite3 chainindex.db "EXPLAIN QUERY PLAN SELECT
   e.event_id,
   tm.height,
   tm.tipset_key_cid,
   e.emitter_addr,
   e.event_index,
   tm.message_cid,
   tm.message_index,
   e.reverted,
   ee.flags,
   ee.key,
   ee.codec,
   ee.value
FROM event e
JOIN tipset_message tm ON e.message_id = tm.message_id
JOIN event_entry ee ON e.event_id = ee.event_id
WHERE tm.tipset_key_cid = 'a' AND e.reverted = false
ORDER BY tm.height DESC, ee.rowid ASC"
QUERY PLAN
|--SEARCH tm USING INDEX idx_tipset_key_cid (tipset_key_cid=?)
|--SEARCH e USING INDEX idx_event_message_id (message_id=?)
|--SEARCH ee USING INDEX event_entry_event_id (event_id=?)
`--USE TEMP B-TREE FOR ORDER BY

[github-actions]

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

[aarshkshah1992]

@rvagg @BigLep @rjan90

Please see https://filecoinproject.slack.com/archives/CP50PPW2X/p1726327482392919

[aarshkshah1992]

These changes will be reverted as the final outcome of that discussion.

[rvagg]

oh, so if you set it anything less than 2880 then you're going to have hashes building up? maybe setting it to less than 2880 should be an error on startup if this is necessary, alternatively convert this to minutes and delete more fine-grained?

[aarshkshah1992]

@rvagg Filed #12465 so we can do it as a follow up. It's a good first issue for a new contributor.

[rvagg]

If we're deferring support then then I think it should be an error to set it to anything 0>x<2880, this is going to be a surprise for anyone not info logging and watching their logs

[aarshkshah1992]

@rvagg This is done. Have added better docs for this param and we now throw an error if it's less than 2880 with an informative error message.