Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sequencer L2 block parallel processing improvements #3604

Merged
merged 21 commits into from
Apr 30, 2024
Merged

Sequencer L2 block parallel processing improvements #3604

merged 21 commits into from
Apr 30, 2024

Conversation

agnusmor
Copy link
Contributor

What does this PR do?

  • Implements L2 block reorg mechanism when processing L2 blocks in parallel. In case an unexpected executor error happens when processing L2 block (e.g. OOC) the sequencer handles the "reorg" of the L2 block to reprocess it syncing the stateroot of the tx-by-tx process and the L2 block process, to ensure txs of that L2 block are executed using the same SR and therefore have the same execution result in both processes
  • Removes waiting to close wip batch to open a new one. Now sequencer can work with 2 batches in parallel, one that is the wip batch (tx-by-tx process) and other that is being processed (L2 blocks) and stored. This allows to increase sequencer performance.
  • Added new config parameter that allows specifiy an interval time that when reached syncs stateroot of the "tx-by-tx" process and "L2 block" process.

[Sequencer.Finalizer]
StateRootSyncInterval = "3600s"

Reviewers

Main reviewers:

@ToniRamirezM
@dpunish3r

@agnusmor
wip
db0fd6b
@agnusmor
first implementation of parallel sequencer optmizations and L2 block …
fdee8f8 
…reorg management
@agnusmor
Close sipBatch (if needed) when processing reorg. Halt when 2 consecu…
8fa6583 
…ties reorgs (same L2 block)
@agnusmor
Return error when reserved counters overflow on l2 block process. Log…
960744f 
… used/reserved counters when closing wip batch
@agnusmor
Merge branch 'release/v0.6.6' into release/l2block-reorg
62225d5
@agnusmor
added logs to analyze blocking issue when storing L2 block
85b9971
@agnusmor
Fix unlock mutex in addTxTracker. Set wipTx to nil in RestoreTxsPendi…
83ecf89 
…ngToStore
@agnusmor
Merge branch 'release/v0.6.6' into release/l2block-reorg
9c022f9
@agnusmor
Merge branch 'release/v0.6.6' into release/l2block-reorg
b677515
@agnusmor
add high reserved resorces in wipBatch
21bf29f
@agnusmor
Merge branch 'release/v0.6.6' into release/l2block-reorg
2aa0565
@agnusmor
store high reserved counter on statedb.batch table
ea87066
@agnusmor
Return contextId in ProcessBatchV2
98d9f58
@agnusmor
Merge branch 'release/v0.6.7' into release/l2block-reorg
6cfc8fa
@agnusmor
fix synchornizer test
af6bb13
@agnusmor
Set SequentialProcessL2Block to false by default. Update node config …
655a9f0 
…documentation
@agnusmor agnusmor added config cherry-picked Content has been cherry-picked into a higher version branch zkevm-seq-agg labels Apr 29, 2024
@agnusmor agnusmor added this to the v0.6.7 milestone Apr 29, 2024
@agnusmor agnusmor self-assigned this Apr 29, 2024
@cla-bot cla-bot bot added the cla-signed label Apr 29, 2024
@agnusmor agnusmor merged commit 8d5cf96 into release/v0.6.7 Apr 30, 2024
30 checks passed
@agnusmor agnusmor deleted the release/l2block-reorg branch April 30, 2024 10:49
@giskook giskook mentioned this pull request May 18, 2024
Open
agnusmor added a commit that referenced this pull request May 21, 2024
@agnusmor
Sequencer L2 block parallel processing improvements (#3604)
b6f4256 
* wip

* first implementation of parallel sequencer optmizations and L2 block reorg management

* Close sipBatch (if needed) when processing reorg. Halt when 2 consecuties reorgs (same L2 block)

* Return error when reserved counters overflow on l2 block process. Log used/reserved counters when closing wip batch

* added logs to analyze blocking issue when storing L2 block

* Fix unlock mutex in addTxTracker. Set wipTx to nil in RestoreTxsPendingToStore

* add high reserved resorces in wipBatch

* store high reserved counter on statedb.batch table

* Return contextId in ProcessBatchV2

* fix synchornizer test

* Set SequentialProcessL2Block to false by default. Update node config documentation

* fix non-e2e tests

* fix finalizer tests

* remove unused code

* test

* Fix sequencer loadFromPool gofunc. Fix docker compose variables
@agnusmor agnusmor mentioned this pull request May 21, 2024
Merged
agnusmor added a commit that referenced this pull request May 22, 2024
@agnusmor
Sequencer L2 block parallel processing improvements (#3604) (#3637)
a5f9558 
* wip

* first implementation of parallel sequencer optmizations and L2 block reorg management

* Close sipBatch (if needed) when processing reorg. Halt when 2 consecuties reorgs (same L2 block)

* Return error when reserved counters overflow on l2 block process. Log used/reserved counters when closing wip batch

* added logs to analyze blocking issue when storing L2 block

* Fix unlock mutex in addTxTracker. Set wipTx to nil in RestoreTxsPendingToStore

* add high reserved resorces in wipBatch

* store high reserved counter on statedb.batch table

* Return contextId in ProcessBatchV2

* fix synchornizer test

* Set SequentialProcessL2Block to false by default. Update node config documentation

* fix non-e2e tests

* fix finalizer tests

* remove unused code

* test

* Fix sequencer loadFromPool gofunc. Fix docker compose variables
@agnusmor agnusmor mentioned this pull request May 22, 2024
Open
Stefan-Ethernal pushed a commit to 0xPolygon/cdk-validium-node that referenced this pull request May 24, 2024
@agnusmor @Stefan-Ethernal
Sequencer L2 block parallel processing improvements (0xPolygonHermez#…
8dee72b 
…3604)

* wip

* first implementation of parallel sequencer optmizations and L2 block reorg management

* Close sipBatch (if needed) when processing reorg. Halt when 2 consecuties reorgs (same L2 block)

* Return error when reserved counters overflow on l2 block process. Log used/reserved counters when closing wip batch

* added logs to analyze blocking issue when storing L2 block

* Fix unlock mutex in addTxTracker. Set wipTx to nil in RestoreTxsPendingToStore

* add high reserved resorces in wipBatch

* store high reserved counter on statedb.batch table

* Return contextId in ProcessBatchV2

* fix synchornizer test

* Set SequentialProcessL2Block to false by default. Update node config documentation

* fix non-e2e tests

* fix finalizer tests

* remove unused code

* test

* Fix sequencer loadFromPool gofunc. Fix docker compose variables
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ToniRamirezM ToniRamirezM ToniRamirezM approved these changes

@tclemos tclemos Awaiting requested review from tclemos tclemos is a code owner

@ARR552 ARR552 Awaiting requested review from ARR552 ARR552 is a code owner

@joanestebanr joanestebanr Awaiting requested review from joanestebanr joanestebanr is a code owner

@dpunish3r dpunish3r Awaiting requested review from dpunish3r

Assignees

@agnusmor agnusmor

Labels
cherry-picked Content has been cherry-picked into a higher version branch cla-signed config zkevm-seq-agg
Projects
None yet
Milestone
v0.6.7
Development

Successfully merging this pull request may close these issues.
2 participants
@agnusmor @ToniRamirezM