Modularize beacon node backend #4718

eserilev · 2023-09-09T15:51:12Z

Issue Addressed

Proposed Changes

Modularize the beacon node backend to make it easier to add new database implementations

Additional Info

The existing KeyValueStore trait already does some abstraction, however the codebases assumes the KV store is always LevelDB.

I created a BeaconNodeBackend type that implements the KeyValueStore and ItemStore traits. I then replaced all references of LevelDb to the new BeaconNodeBackend type. Within the BeaconNodeBackend type I used the cfg macro which should allow us to switch between different database implementations via config changes.

…dularize-beacon-node-backend

…ackend

eserilev · 2024-01-30T21:19:26Z

I'm currently working on adding Redb to the beacon node backend. Here are some notes I've taken so far on my work

Redb

Theres a few difference's between LevelDB and Redb that I'll describe below

fsync

Info about redb transaction durability can be found here
https://docs.rs/redb/latest/redb/enum.Durability.html#

For now I translate WriteOptions.sync = true to be Durability::Immediate and WriteOptions.sync = false to be Durability::Eventual

We could decide to use Paranoid instead of Immediate, but I think the additional guarantees of Paranoid are probably unecessary for our use case.

Compaction

LevelDB allows for compacting over a range of values. Redb only allows for compacting the full db. It seems like our LevelDB implementation compacts across all keys in the DB. If thats the case we should be all good here.

Tables

Redb introduces the concept of tables. We can either use one table for everything, or spin up tables by column name. I'm currently going down the path of spinning up tables by column name. This causes issues with do_atomically. We may need to make changes to this function to also accept a col: &str as an argument.

…ackend

eserilev · 2024-08-21T05:25:25Z

This PR is in a good state for a review. We've looked at some stats for the redb node running on gown and compared it to another holesky node running leveldb. Db read i/o seems to be on par, and we actually saw lower db write i/o on redb. One thing to pay attention to is compaction. We dug into some logs and didnt see any drastic degredation in performance during compaction for redb. Just note that compaction does lock the db, so its definitely something to keep an eye on.

dapplion · 2024-08-26T12:31:11Z

beacon_node/beacon_chain/src/schema_change/migration_schema_v18.rs

@@ -0,0 +1,117 @@
+use crate::beacon_chain::BeaconChainTypes;


This file and beacon_node/beacon_chain/src/schema_change/migration_schema_v19.rs seem to be added by mistake?

removed those files

dapplion · 2024-08-26T12:34:56Z

beacon_node/store/src/database/leveldb_impl.rs

+    }
+
+    /// Compact all values in the states and states flag columns.
+    pub fn compact(&self) -> Result<(), Error> {


Could add timing metrics for compact, either here on in the caller

beacon_node/store/src/database/interface.rs

dapplion · 2024-08-26T12:44:05Z

beacon_node/store/src/database/redb_impl.rs

+            let table = read_txn.open_table(table_definition)?;
+            table.range(from..)?.map(|res| {
+                let (k, _) = res?;
+                K::from_bytes(k.value())


read metrics should cover the iterator too

dapplion · 2024-08-26T13:43:43Z

beacon_node/beacon_chain/src/light_client_server_cache.rs

+            column,
+            &start_period.to_le_bytes(),
+            move |sync_committee_bytes, _| {
+                let Ok(sync_committee_period) = u64::from_ssz_bytes(sync_committee_bytes) else {


Is it safe to silence this error?

It's probably safe to silence this error, but I'm now logging the error just in case
8ed2ff4

dapplion · 2024-08-26T13:46:02Z

beacon_node/store/src/database/redb_impl.rs

+    pub fn sync(&self) -> Result<(), Error> {
+        self.put_bytes_sync("sync", b"sync", b"sync")
+    }


What's the purpose of this method?

it gets called during the hot -> cold db migration.

// Flush to disk all the states that have just been migrated to the cold store.

My guess is that we do a quick write to the cold db just to ensure that there aren't any hanging state changes before starting the migration? sync is a dummy table that stores a single, non useful, key/value pair.

beacon_node/store/src/database/redb_impl.rs

dapplion · 2024-08-26T13:48:05Z

beacon_node/store/src/lib.rs

+    PutKeyValue(String, Vec<u8>, Vec<u8>),
+    DeleteKey(String, Vec<u8>),


Document the tuple

dapplion · 2024-08-28T07:51:21Z

beacon_node/store/src/database/leveldb_impl.rs

+                key.matches_column(column) && predicate(trimmed_key, trimmed_start_key)
+            })
+            .map(move |(bytes_key, value)| {
+                let key = bytes_key.remove_column_variable(column).ok_or_else(|| {


You should add the metric here and increase DISK_DB_READ_BYTES with value to have the same meaning as the the usage with the per-item gettter

dapplion · 2024-08-28T07:52:44Z

beacon_node/store/src/database/leveldb_impl.rs

+        Ok(Box::new(
+            iter.take_while(move |key| key.matches_column(column))
+                .map(move |bytes_key| {
+                    metrics::inc_counter_vec(&metrics::DISK_DB_READ_COUNT, &[column.into()]);


I think it's not ideal to count reads of keys (this function) as reads of entries (other uses of DISK_DB_READ_COUNT). If this iterators just read keys, maybe we should use a different metric DISK_DB_KEY_READ_COUNT or similar

yep makes sense, I've added separate metrics for db key read

…serilev/lighthouse into modularize-beacon-node-backend

eserilev · 2024-08-30T03:50:53Z

I've opened a separate issue around temp state cleanup performance for Redb. I'm going to handle that in a separate PR if thats ok with you guys

#6332

eserilev · 2024-09-01T21:39:25Z

Compaction in redb v2.1.2 is craaazy slow: cberner/redb#852

For now lets just make sure we stay on 2.1.1 until the issue is resolved

…serilev/lighthouse into modularize-beacon-node-backend

…dularize-beacon-node-backend

eserilev added 8 commits August 31, 2023 19:26

add interface

0487b33

Merge branch 'unstable' of https://github.com/sigp/lighthouse into mo…

4b48ef8

…dularize-beacon-node-backend

level db interface updates

ec84723

level db interface updates

86b820e

starting the split

846b213

remove leveldb references

a885c62

get test cases to pass

b8636d1

refactor and get test cases to pass

7177f22

eserilev changed the title ~~[WIP] Modularize beacon node backend~~ Modularize beacon node backend Sep 9, 2023

generalize key iter

d86eaf3

michaelsproul mentioned this pull request Sep 29, 2023

Lighthouse space usage - Increasing dramastically from 87GB to 430GB in 7 days #4773

Closed

jimmygchen added ready-for-review The code is ready for review database labels Oct 18, 2023

eserilev added 10 commits January 30, 2024 14:45

resolve merge conflicts

d6c4971

rename impl to LevelDB

2ee7279

initial work

cc1dcf4

write option

b6a2823

cfg

3881da1

merge

1d39785

redb db impl

04011f3

redb

e2ecb41

durability and atomicity

323e9a9

Merge branch 'beacon-node-backend-redb' into modularize-beacon-node-b…

e2a9f7c

…ackend

eserilev added 6 commits January 31, 2024 08:36

remove savepoint

ad24ec9

Merge branch 'beacon-node-backend-redb' into modularize-beacon-node-b…

3e557d6

…ackend

working on getting full_participation_no_skips test to pass

39c6b83

test case passes

f718f9f

update

41700d6

use rw lock

04eada1

remove println

ced1189

dapplion reviewed Aug 26, 2024

View reviewed changes

eserilev added 9 commits August 26, 2024 09:04

remove extraneous migration schemas and comment tuples

ec7059b

add compaction metrics

ad7db41

log from_ssz_bytes error in lc server

8ed2ff4

add additional metrics

63b15b1

linting

fc2e412

fixbroken test

74e1c07

metrics

72b6381

fix build error

627f013

small revert

6237531

dapplion reviewed Aug 28, 2024

View reviewed changes

eserilev added 4 commits August 28, 2024 19:02

resolve merge conflicts:

df59d23

Merge branch 'modularize-beacon-node-backend' of https://github.com/e…

d97aeaf

…serilev/lighthouse into modularize-beacon-node-backend

update metrics

0bab63c

fix redb

06490d4

eserilev mentioned this pull request Aug 30, 2024

Optimize Redb temporary state cleanup #6332

Open

eserilev added 10 commits September 9, 2024 10:58

fix conflicts, add TODOS

89866f2

Merge branch 'modularize-beacon-node-backend' of https://github.com/e…

f2c514a

…serilev/lighthouse into modularize-beacon-node-backend

remove todos

7428536

fix audits

910f45d

conflicts

cf3b056

resolve conflicts, add some TODOs

b589e2d

Merge branch 'unstable' of https://github.com/sigp/lighthouse into mo…

e3cdca5

…dularize-beacon-node-backend

update redb version to 2.1.3

282aa16

get tests to pass

807137a

Merge branch 'unstable' of https://github.com/sigp/lighthouse into mo…

4b7a4ff

…dularize-beacon-node-backend

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modularize beacon node backend #4718

Modularize beacon node backend #4718

eserilev commented Sep 9, 2023 •

edited

Loading

eserilev commented Jan 30, 2024 •

edited

Loading

eserilev commented Aug 21, 2024 •

edited

Loading

dapplion Aug 26, 2024

eserilev Aug 26, 2024

dapplion Aug 26, 2024

eserilev Aug 26, 2024 •

edited

Loading

dapplion Aug 26, 2024

eserilev Aug 26, 2024

dapplion Aug 26, 2024

eserilev Aug 26, 2024

dapplion Aug 26, 2024

eserilev Aug 26, 2024

dapplion Aug 26, 2024

eserilev Aug 26, 2024

dapplion Aug 28, 2024

eserilev Aug 29, 2024

dapplion Aug 28, 2024

eserilev Aug 29, 2024

eserilev commented Aug 30, 2024

eserilev commented Sep 1, 2024

		PutKeyValue(String, Vec<u8>, Vec<u8>),
		DeleteKey(String, Vec<u8>),

Modularize beacon node backend #4718

Are you sure you want to change the base?

Modularize beacon node backend #4718

Conversation

eserilev commented Sep 9, 2023 • edited Loading

Issue Addressed

Proposed Changes

Additional Info

eserilev commented Jan 30, 2024 • edited Loading

Redb

fsync

Compaction

Tables

eserilev commented Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eserilev Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eserilev commented Aug 30, 2024

eserilev commented Sep 1, 2024

eserilev commented Sep 9, 2023 •

edited

Loading

eserilev commented Jan 30, 2024 •

edited

Loading

eserilev commented Aug 21, 2024 •

edited

Loading

eserilev Aug 26, 2024 •

edited

Loading