Pullupstream/v0.5.8 (#74)

* fix migration batch wip field. fix sequence when no wip batch (0xPolygonHermez#3126)

* fix Leaves smtproof (0xPolygonHermez#3127)

* fix Leaves smtproof

* pointer protection

* fix unit test

* Add l2block log dump when storing error. Refactor ProcessBatchV2 logs (0xPolygonHermez#3129)

* add l2block log dump when storing error. refactor ProcessBatchV2 logs

* add AddL2Block log

* fix index AddL2Block log

* use aux variable to avoid reusing poiner

---------

Co-authored-by: Toni Ramírez <toni@polygon.technology>

* fix panic etherman (0xPolygonHermez#3133)

* Fix GetL2Hash function. Use tx L2 hash returned by the executor when storing L2Block. Logs changes (0xPolygonHermez#3134)

* fix l2 hash when to is nill

* get l2 hash from executor

* fix use tx L2 hash returned by the executor when storing L2Block. Logs changes

* fix linter

* add L2block tracking number to process tx logs

* fix ProcessBatchV2 log format

* fix UTs. set useMainExecGenerated to true

* set prover image to v4.0.0-RC30

* test setting useMainExecGenerated = false

* fix independent_test

* fix non-e2e tests

---------

Co-authored-by: Toni Ramírez <toni@polygon.technology>

* remove l2 tx hash computation and use data stored in the state returned by the executor (0xPolygonHermez#3139)

* etrog: fix permissionless errors (0xPolygonHermez#3140)

* if got an error getting lastBlock produce a SIGSEGV

* reduce info logs

* fix unittest

* Fix missing batch bookmark (0xPolygonHermez#3122)

* fix missingBookMark

* fix missingBookMark

* fixes

* fix

* fix

* fix seq-sender config

* fixes

* doc

* fix wipL2Block deltaTimestamp (0xPolygonHermez#3142)

* improve open new wipL2Block logs

* fix wipL2Block deltaTimestamp

* ensure tx order in stream (0xPolygonHermez#3144)

* fix wipL2Block stateRoot (0xPolygonHermez#3145)

* fix panic NewStackTrie (0xPolygonHermez#3146)

* fix sync initalization for etrog (0xPolygonHermez#3147)

* fix wipL2block imStateRoot (0xPolygonHermez#3148)

* fix tracer to work by block stateroot instead of by tx (0xPolygonHermez#3150)

* update prover image (0xPolygonHermez#3151)

* if a trusted batch is empty and WIP just create the entry in state.batch (0xPolygonHermez#3152)

* fix trace for tx index greater than 0 (0xPolygonHermez#3153)

* add cardona.zip (0xPolygonHermez#3154)

* Fix default value when creating transaction.used_sha256_hashes field in the pool (0xPolygonHermez#3156)

* set DEFAULT 0 when creatingtransaction.used_sha256_hashes field in the pool

* update prover image to v4.0.0

* add cardona testnet (0xPolygonHermez#2909) (0xPolygonHermez#3155)

* add cardona testnet (0xPolygonHermez#2909)
* update config
* update cardona genesis

---------

Co-authored-by: Thiago Coimbra Lemos <tclemos@users.noreply.github.com>

* new fields in stream (0xPolygonHermez#3149)

* new fields in stream

* update test

* remove unused constant

* add UpgradeEtrogBatchNumber handling to stream

* doc

* add log to test

* add log to test

* add protection

* add check

* fix

* fix

* set stateManagerPurge to false in Cardona (0xPolygonHermez#3158)

* synchronizer: fix case emtpy batch and unittest (0xPolygonHermez#3159)

* fix case emtpy batch and unittest

* fix error in conversion

* fix unittest

* update prover image to v4.0.1 (0xPolygonHermez#3160)

* update prover image to v4.0.2 (0xPolygonHermez#3162)

* Add a note in deployment instruction to restart rpc after forkId 7 (0xPolygonHermez#3163)

* note to restart zkevm-rpc

* update comment

* change position of point 10

* add point 11

* add lines

* add breakline

* add lines

* add line

* Sync old network compatibility (0xPolygonHermez#3166)

* Sync old network compatibility

* logs

* synchronizer: unittest and remove pool_reorg  (0xPolygonHermez#3111) (0xPolygonHermez#3168)

* removed reorgPool call
* add test for l1 sequenced batches
* removed panic after halt call as PR comments ask for

* fix parentHash when storing a new L2 block (0xPolygonHermez#3171)

* Fix/rom error (0xPolygonHermez#3177)

* fix rom error

* linter

* fix

* Cherrypick/0xPolygonHermez#3179 geth version v1.13.11 (0xPolygonHermez#3180)

* new geth version v1.13.11

* Add sanity check: last L2 block timestamp in the sequence is also L1BlockTimestampMargin seconds behind "now" (0xPolygonHermez#3181)

* add sanity check L1BlockTimestampMargin vs now

* Update prover image to v4.0.4

* synchronizer: update fromTrusted fix cache update (0xPolygonHermez#3169)

* fix cache update and sync from Trusted of closed batch

* Feature/3173 sync trusted state in networks previous to  ETROG (0xPolygonHermez#3175)

* Fix/3164 eth syncing dont return expected values (0xPolygonHermez#3165)

* fix im state root in stream (0xPolygonHermez#3192)

* sync: fix getting URL from contract in compatibility mode, and set sequential by default (0xPolygonHermez#3191)

* fix getting URL from contract in compatibility mode
* synchronizer configuration  by default use `sequential` mode

* Fix isSynced function (0xPolygonHermez#3188)

* fix isSynced

* fix linter

* fix logs

* halt on sanity check error

* fix l2 block gas price to use value returned by the executor (0xPolygonHermez#3186) (0xPolygonHermez#3190)

* allow rpc to access the ForkID directly from DB (0xPolygonHermez#3196)

* allow rpc to access the ForkID directly from DB

* update docs

* allow rpc to access the ForkID directly from DB (0xPolygonHermez#3197)

* allow rpc to access the ForkID directly from DB

* update docs

* fix migration down; check for greater forkIDs instead of the next +1 (0xPolygonHermez#3182)

* fix state storage config initialization

* add tests and fix get forkID by block number

* fix test TestGetForkIDByBlockNumber

---------

Co-authored-by: agnusmor <agnusmor@gmail.com>

* Fix/3200 l1inforoot error on reorg (0xPolygonHermez#3201)

* L1InfooTree cache is calculated when needed and is discard when a reorg happens

* Fix/3203 panic if batch process fail and must be closed (0xPolygonHermez#3206)

* synchronizer avoid panic if process trusted batch fails

* Remove L1SynchronizationMode parameter from environments config files (it will use default "sequential" value) (0xPolygonHermez#3210)

* allow GetL2TxHashByTxHash to return null for txs prior to Etrog (0xPolygonHermez#3209)

* Fix L2 block gas limit to 2**50 (1125899906842624) (0xPolygonHermez#3219)

* set l2 block gas limit to 2**50 (1125899906842624)

* limit L2 block gasLimit in StoreL2Block function

* change 2^64-1 for an esmitation of blocks (0xPolygonHermez#3220)

* removed restart RPC on deploy instructions (0xPolygonHermez#3221)

* Cherry pick/3222 (0xPolygonHermez#3224)

* fix stream

* fix stream

* fix stream

* fix stream

* linter

* add retries in is Synced function when last virtual batch is greater than last SC sequenced batcj (0xPolygonHermez#3225)

* Fix/3233 the etrog event UpdateEtrogSequence doesnt store the GER in batch table (0xPolygonHermez#3234)

* remove state.batch section from environments config files (0xPolygonHermez#3230)

* get GER for l2block on Etrog (0xPolygonHermez#3238)

* Cherry-pick 0xPolygonHermez#3240, 0xPolygonHermez#3137: Sequence batch only when sanity check is done. Do fatal for some finalizer halt events. Several fixes. Optmize purge pool (0xPolygonHermez#3241)

* sequence batch only when sanity check is done. Do fatal for some halt events. Several fixes

* feat: optimize purge txn pool query (0xPolygonHermez#3137)

* feat: optimize purge txn pool query

* fix: add new method to state interface

* feat: generate new state mocks

* fix merge

* fix %w

---------

Co-authored-by: Idris Hanafi <ifh101@gmail.com>

* fix

* docs update

* fix

---------

Co-authored-by: agnusmor <100322135+agnusmor@users.noreply.github.com>
Co-authored-by: Alonso Rodriguez <ARR552@users.noreply.github.com>
Co-authored-by: Toni Ramírez <toni@polygon.technology>
Co-authored-by: Thiago Coimbra Lemos <tclemos@users.noreply.github.com>
Co-authored-by: Joan Esteban <129153821+joanestebanr@users.noreply.github.com>
Co-authored-by: Toni Ramírez <58293609+ToniRamirezM@users.noreply.github.com>
Co-authored-by: agnusmor <agnusmor@gmail.com>
Co-authored-by: Idris Hanafi <ifh101@gmail.com>

db/migrations/state/0016.sql

@@ -0,0 +1,7 @@
+-- +migrate Up
+ALTER TABLE state.batch
+    ADD COLUMN IF NOT EXISTS checked BOOLEAN NOT NULL DEFAULT TRUE;
+
+-- +migrate Down
+ALTER TABLE state.batch
+    DROP COLUMN IF EXISTS checked;

db/migrations/state/0016_test.go

@@ -0,0 +1,64 @@
+package migrations_test
+
+import (
+	"database/sql"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+type migrationTest0016 struct{}
+
+func (m migrationTest0016) InsertData(db *sql.DB) error {
+	const insertBatch0 = `
+		INSERT INTO state.batch (batch_num, global_exit_root, local_exit_root, acc_input_hash, state_root, timestamp, coinbase, raw_txs_data, forced_batch_num, wip) 
+		VALUES (0,'0x0000', '0x0000', '0x0000', '0x0000', now(), '0x0000', null, null, true)`
+
+	// insert batch
+	_, err := db.Exec(insertBatch0)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (m migrationTest0016) RunAssertsAfterMigrationUp(t *testing.T, db *sql.DB) {
+	var result int
+
+	// Check column checked exists in state.batch table
+	const getCheckedColumn = `SELECT count(*) FROM information_schema.columns WHERE table_name='batch' and column_name='checked'`
+	row := db.QueryRow(getCheckedColumn)
+	assert.NoError(t, row.Scan(&result))
+	assert.Equal(t, 1, result)
+
+	const insertBatch0 = `
+		INSERT INTO state.batch (batch_num, global_exit_root, local_exit_root, acc_input_hash, state_root, timestamp, coinbase, raw_txs_data, forced_batch_num, wip, checked) 
+		VALUES (1,'0x0001', '0x0001', '0x0001', '0x0001', now(), '0x0001', null, null, true, false)`
+
+	// insert batch 1
+	_, err := db.Exec(insertBatch0)
+	assert.NoError(t, err)
+
+	const insertBatch1 = `
+		INSERT INTO state.batch (batch_num, global_exit_root, local_exit_root, acc_input_hash, state_root, timestamp, coinbase, raw_txs_data, forced_batch_num, wip, checked) 
+		VALUES (2,'0x0002', '0x0002', '0x0002', '0x0002', now(), '0x0002', null, null, false, true)`
+
+	// insert batch 2
+	_, err = db.Exec(insertBatch1)
+	assert.NoError(t, err)
+}
+
+func (m migrationTest0016) RunAssertsAfterMigrationDown(t *testing.T, db *sql.DB) {
+	var result int
+
+	// Check column wip doesn't exists in state.batch table
+	const getCheckedColumn = `SELECT count(*) FROM information_schema.columns WHERE table_name='batch' and column_name='checked'`
+	row := db.QueryRow(getCheckedColumn)
+	assert.NoError(t, row.Scan(&result))
+	assert.Equal(t, 0, result)
+}
+
+func TestMigration0016(t *testing.T) {
+	runMigrationTest(t, 16, migrationTest0016{})
+}

docs/diff/diff.md

@@ -50,7 +50,7 @@ diff -ruN \
 -x "coverage.out" \
 -x "*datastream.db*" \
 ${PATH_TO_ZKEVM_NODE_REPO} . | \
-diff2html -i stdin -s side -t "zkEVM node vs CDK validium node</br><h2>zkevm-node version: v0.5.6<h2/>" \
+diff2html -i stdin -s side -t "zkEVM node vs CDK validium node</br><h2>zkevm-node version: v0.5.8<h2/>" \
 -F ./docs/diff/diff.html
 ```
 

etherman/simulated.go

@@ -211,8 +211,8 @@ func NewSimulatedEtherman(cfg Config, auth *bind.TransactOpts, daBackend dataAva
 		log.Error("error: ", err)
 		return nil, nil, common.Address{}, nil, err
 	}
-
 	client.Commit()
+
 	c := &Client{
 		EthClient:             client.Client(),
 		ZkEVM:                 trueZkevm,

go.mod

@@ -38,7 +38,6 @@ require (
 
 require (
 	dario.cat/mergo v1.0.0 // indirect
-	github.com/0xPolygon/agglayer v0.0.1 // indirect
 	github.com/DataDog/zstd v1.5.2 // indirect
 	github.com/Microsoft/go-winio v0.6.1 // indirect
 	github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
@@ -173,6 +172,7 @@ require (
 )
 
 require (
+	github.com/0xPolygon/agglayer v0.0.1
 	github.com/0xPolygon/cdk-data-availability v0.0.5
 	github.com/fatih/color v1.16.0
 	github.com/joho/godotenv v1.5.1

go.sum

@@ -41,8 +41,6 @@ dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 github.com/0xPolygon/agglayer v0.0.1 h1:J6/DUo9rNUncDifquanouRCo2g7g069yvz0aFtu7R/8=
 github.com/0xPolygon/agglayer v0.0.1/go.mod h1:UYp5O8THULoXVrUfzkRjVBzxHR5DxBdUN/Iq0EgxNxM=
-github.com/0xPolygon/cdk-data-availability v0.0.4 h1:FFG6rkLnkgknhMknxXFCaQLZpZXe/EHL3If+qRPieXk=
-github.com/0xPolygon/cdk-data-availability v0.0.4/go.mod h1:aGwqHiJhL+mJbdepl3s58wsY18EuViDa9vZCpPuIYGw=
 github.com/0xPolygon/cdk-data-availability v0.0.5 h1://vg1oR/5tw2XfEIorpP+wIxLfNUmoKrdmX8YZvBKX4=
 github.com/0xPolygon/cdk-data-availability v0.0.5/go.mod h1:aGwqHiJhL+mJbdepl3s58wsY18EuViDa9vZCpPuIYGw=
 github.com/0xPolygonHermez/zkevm-data-streamer v0.1.18 h1:InqeTcHrNbfj1OUfn2aFplFay7ibd7KhYqvmMZYZfn0=
@@ -174,7 +172,6 @@ github.com/deckarep/golang-set/v2 v2.1.0 h1:g47V4Or+DUdzbs8FxCCmgb6VYd+ptPAngjM6
 github.com/deckarep/golang-set/v2 v2.1.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
 github.com/decred/dcrd/crypto/blake256 v1.0.1 h1:7PltbUIQB7u/FfZ39+DGa/ShuMyJ5ilcvdfma9wOH6Y=
 github.com/decred/dcrd/crypto/blake256 v1.0.1/go.mod h1:2OfgNZ5wDpcsFmHmCK5gZTPcCXqlm2ArzUIkw9czNJo=
-github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0/go.mod h1:DZGJHZMqrU4JJqFAWUS2UO1+lbSKsdiOoYi9Zzey7Fc=
 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 h1:8UrgZ3GkP4i/CLijOJx79Yu+etlyjdBU4sfcs2WYQMs=
 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0/go.mod h1:v57UDF4pDQJcEfFUCRop3lJL149eHGSe9Jvczhzjo/0=
 github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4=
@@ -218,13 +215,11 @@ github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4Nij
 github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
 github.com/fjl/memsize v0.0.0-20190710130421-bcb5799ab5e5 h1:FtmdgXiUlNeRsoNMFlKLDt+S+6hbjVMEW6RGQ7aUf7c=
 github.com/fjl/memsize v0.0.0-20190710130421-bcb5799ab5e5/go.mod h1:VvhXpOYNQvB+uIk2RvXzuaQtkQJzzIx6lSBe1xv7hi0=
-github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY=
-github.com/frankban/quicktest v1.14.4/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
+github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
+github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/fsnotify/fsnotify v1.5.4/go.mod h1:OVB6XrOHzAwXMpEM7uPOzcehqUV2UqJxmVXmkdnm1bU=
-github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
-github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
 github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
 github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
 github.com/gavv/httpexpect v2.0.0+incompatible/go.mod h1:x+9tiU1YnrOvnB725RkpoLv1M62hOWzwo5OXotisrKc=
@@ -699,7 +694,6 @@ github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
 github.com/ryanuber/columnize v2.1.0+incompatible/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
-github.com/sagikazarmark/locafero v0.3.0/go.mod h1:w+v7UsPNFwzF1cHuOajOOzoq4U7v/ig1mpRjqV+Bu1U=
 github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6keLGt6kNQ=
 github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4=
 github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
@@ -739,7 +733,6 @@ github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8=
 github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
 github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
-github.com/spf13/cast v1.5.1/go.mod h1:b9PdjNptOpzXr7Rq1q9gJML/2cdGQAo69NKzQ10KN48=
 github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0=
 github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
 github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
@@ -751,7 +744,6 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
 github.com/spf13/viper v1.8.1/go.mod h1:o0Pch8wJ9BVSWGQMbra6iw0oQ5oktSIBaujf1rJH9Ns=
-github.com/spf13/viper v1.17.0/go.mod h1:BmMMMLQXSbcHK6KAOiFLz0l5JHrU89OdIRHvsk0+yVI=
 github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ=
 github.com/spf13/viper v1.18.2/go.mod h1:EKmWIqdnk5lOcmR72yw6hS+8OPYcwD0jteitLMVB+yk=
 github.com/status-im/keycard-go v0.2.0 h1:QDLFswOQu1r5jsycloeQh3bVU8n/NatHHaZobtDnDzA=

pool/interfaces.go

@@ -39,6 +39,7 @@ type storage interface {
 	GetAllAddressesBlocked(ctx context.Context) ([]common.Address, error)
 	MinL2GasPriceSince(ctx context.Context, timestamp time.Time) (uint64, error)
 	policy
+	GetEarliestProcessedTx(ctx context.Context) (common.Hash, error)
 }
 
 type stateInterface interface {

pool/pgpoolstorage/pgpoolstorage.go

@@ -790,3 +790,23 @@ func (p *PostgresPoolStorage) GetAllAddressesBlocked(ctx context.Context) ([]com
 
 	return addrs, nil
 }
+
+// GetEarliestProcessedTx gets the earliest processed tx from the pool. Mainly used for cleanup
+func (p *PostgresPoolStorage) GetEarliestProcessedTx(ctx context.Context) (common.Hash, error) {
+	const getEarliestProcessedTxnFromTxnPool = `SELECT hash
+	FROM pool.transaction
+	WHERE
+		status = 'selected'
+	ORDER BY received_at ASC
+	LIMIT 1`
+
+	var txnHash string
+	err := p.db.QueryRow(ctx, getEarliestProcessedTxnFromTxnPool).Scan(&txnHash)
+	if errors.Is(err, pgx.ErrNoRows) {
+		return common.Hash{}, nil
+	} else if err != nil {
+		return common.Hash{}, err
+	}
+
+	return common.HexToHash(txnHash), nil
+}

sequencer/batch.go

@@ -33,6 +33,32 @@ func (w *Batch) isEmpty() bool {
 	return w.countOfL2Blocks == 0
 }
 
+// processBatchesPendingtoCheck performs a sanity check for batches closed but pending to be checked
+func (f *finalizer) processBatchesPendingtoCheck(ctx context.Context) {
+	notCheckedBatches, err := f.stateIntf.GetNotCheckedBatches(ctx, nil)
+	if err != nil && err != state.ErrNotFound {
+		log.Fatalf("failed to get batches not checked, error: ", err)
+	}
+
+	if len(notCheckedBatches) == 0 {
+		return
+	}
+
+	log.Infof("executing sanity check for not checked batches")
+
+	prevBatchNumber := notCheckedBatches[0].BatchNumber - 1
+	prevBatch, err := f.stateIntf.GetBatchByNumber(ctx, prevBatchNumber, nil)
+	if err != nil {
+		log.Fatalf("failed to get batch %d, error: ", prevBatchNumber, err)
+	}
+	oldStateRoot := prevBatch.StateRoot
+
+	for _, notCheckedBatch := range notCheckedBatches {
+		_, _ = f.batchSanityCheck(ctx, notCheckedBatch.BatchNumber, oldStateRoot, notCheckedBatch.StateRoot)
+		oldStateRoot = notCheckedBatch.StateRoot
+	}
+}
+
 // setWIPBatch sets finalizer wip batch to the state batch passed as parameter
 func (f *finalizer) setWIPBatch(ctx context.Context, wipStateBatch *state.Batch) (*Batch, error) {
 	// Retrieve prevStateBatch to init the initialStateRoot of the wip batch
@@ -89,7 +115,7 @@ func (f *finalizer) initWIPBatch(ctx context.Context) {
 	// Get the last batch in trusted state
 	lastStateBatch, err := f.stateIntf.GetBatchByNumber(ctx, lastBatchNum, nil)
 	if err != nil {
-		log.Fatalf("failed to get last batch, error: %v", err)
+		log.Fatalf("failed to get last batch %d, error: %v", lastBatchNum, err)
 	}
 
 	isClosed := !lastStateBatch.WIP
@@ -98,7 +124,7 @@ func (f *finalizer) initWIPBatch(ctx context.Context) {
 
 	if isClosed { //if the last batch is close then open a new wip batch
 		if lastStateBatch.BatchNumber+1 == f.cfg.HaltOnBatchNumber {
-			f.Halt(ctx, fmt.Errorf("finalizer reached stop sequencer on batch number: %d", f.cfg.HaltOnBatchNumber))
+			f.Halt(ctx, fmt.Errorf("finalizer reached stop sequencer on batch number: %d", f.cfg.HaltOnBatchNumber), false)
 		}
 
 		f.wipBatch, err = f.openNewWIPBatch(ctx, lastStateBatch.BatchNumber+1, lastStateBatch.StateRoot)
@@ -126,14 +152,14 @@ func (f *finalizer) finalizeWIPBatch(ctx context.Context, closeReason state.Clos
 	prevTimestamp := f.wipL2Block.timestamp
 	prevL1InfoTreeIndex := f.wipL2Block.l1InfoTreeExitRoot.L1InfoTreeIndex
 
-	// Close the wip L2 block if it has transactions, if not we keep it open to store it in the new wip batch
+	// Close the wip L2 block if it has transactions, otherwise we keep the wip L2 block to store it in the new wip batch
 	if !f.wipL2Block.isEmpty() {
 		f.closeWIPL2Block(ctx)
 	}
 
 	err := f.closeAndOpenNewWIPBatch(ctx, closeReason)
 	if err != nil {
-		f.Halt(ctx, fmt.Errorf("failed to create new WIP batch, error: %v", err))
+		f.Halt(ctx, fmt.Errorf("failed to create new WIP batch, error: %v", err), true)
 	}
 
 	// If we have closed the wipL2Block then we open a new one
@@ -142,8 +168,18 @@ func (f *finalizer) finalizeWIPBatch(ctx context.Context, closeReason state.Clos
 	}
 }
 
-// closeAndOpenNewWIPBatch closes the current batch and opens a new one, potentially processing forced batches between the batch is closed and the resulting new empty batch
+// closeAndOpenNewWIPBatch closes the current batch and opens a new one, potentially processing forced batches between the batch is closed and the resulting new wip batch
 func (f *finalizer) closeAndOpenNewWIPBatch(ctx context.Context, closeReason state.ClosingReason) error {
+	f.nextForcedBatchesMux.Lock()
+	processForcedBatches := len(f.nextForcedBatches) > 0
+	f.nextForcedBatchesMux.Unlock()
+
+	// If we will process forced batches after we close the wip batch then we must close the current wip L2 block,
+	// since the processForcedBatches function needs to create new L2 blocks (cannot "reuse" the current wip L2 block if it's empty)
+	if processForcedBatches {
+		f.closeWIPL2Block(ctx)
+	}
+
 	// Wait until all L2 blocks are processed by the executor
 	startWait := time.Now()
 	f.pendingL2BlocksToProcessWG.Wait()
@@ -170,11 +206,7 @@ func (f *finalizer) closeAndOpenNewWIPBatch(ctx context.Context, closeReason sta
 	// Reprocess full batch as sanity check
 	if f.cfg.SequentialBatchSanityCheck {
 		// Do the full batch reprocess now
-		_, err := f.batchSanityCheck(ctx, f.wipBatch.batchNumber, f.wipBatch.initialStateRoot, f.wipBatch.finalStateRoot)
-		if err != nil {
-			// There is an error reprocessing the batch. We halt the execution of the Sequencer at this point
-			return fmt.Errorf("halting sequencer because of error reprocessing full batch %d (sanity check), error: %v ", f.wipBatch.batchNumber, err)
-		}
+		_, _ = f.batchSanityCheck(ctx, f.wipBatch.batchNumber, f.wipBatch.initialStateRoot, f.wipBatch.finalStateRoot)
 	} else {
 		// Do the full batch reprocess in parallel
 		go func() {
@@ -183,15 +215,15 @@ func (f *finalizer) closeAndOpenNewWIPBatch(ctx context.Context, closeReason sta
 	}
 
 	if f.wipBatch.batchNumber+1 == f.cfg.HaltOnBatchNumber {
-		f.Halt(ctx, fmt.Errorf("finalizer reached stop sequencer on batch number: %d", f.cfg.HaltOnBatchNumber))
+		f.Halt(ctx, fmt.Errorf("finalizer reached stop sequencer on batch number: %d", f.cfg.HaltOnBatchNumber), false)
 	}
 
 	// Metadata for the next batch
 	stateRoot := f.wipBatch.finalStateRoot
 	lastBatchNumber := f.wipBatch.batchNumber
 
 	// Process forced batches
-	if len(f.nextForcedBatches) > 0 {
+	if processForcedBatches {
 		lastBatchNumber, stateRoot = f.processForcedBatches(ctx, lastBatchNumber, stateRoot)
 		// We must init/reset the wip L2 block from the state since processForcedBatches can created new L2 blocks
 		f.initWIPL2Block(ctx)
@@ -315,12 +347,13 @@ func (f *finalizer) batchSanityCheck(ctx context.Context, batchNum uint64, initi
 		// Log batch detailed info
 		log.Errorf("batch %d sanity check error: initialStateRoot: %s, expectedNewStateRoot: %s", batch.BatchNumber, initialStateRoot, expectedNewStateRoot)
 		for i, rawL2block := range rawL2Blocks.Blocks {
+			log.Infof("block[%d], txs: %d, deltaTimestamp: %d, l1InfoTreeIndex: %d", i, len(rawL2block.Transactions), rawL2block.DeltaTimestamp, rawL2block.IndexL1InfoTree)
 			for j, rawTx := range rawL2block.Transactions {
-				log.Infof("batch %d, block position: %d, tx position: %d, tx hash: %s", batch.BatchNumber, i, j, rawTx.Tx.Hash())
+				log.Infof("block[%d].tx[%d]: %s, egpPct: %d", batch.BatchNumber, i, j, rawTx.Tx.Hash(), rawTx.EfficiencyPercentage)
 			}
 		}
 
-		f.Halt(ctx, fmt.Errorf("batch sanity check error. Check previous errors in logs to know which was the cause"))
+		f.Halt(ctx, fmt.Errorf("batch sanity check error. Check previous errors in logs to know which was the cause"), false)
 	}
 
 	log.Debugf("batch %d sanity check: initialStateRoot: %s, expectedNewStateRoot: %s", batchNum, initialStateRoot, expectedNewStateRoot)
@@ -404,6 +437,13 @@ func (f *finalizer) batchSanityCheck(ctx context.Context, batchNum uint64, initi
 		return nil, ErrStateRootNoMatch
 	}
 
+	err = f.stateIntf.UpdateBatchAsChecked(ctx, batch.BatchNumber, nil)
+	if err != nil {
+		log.Errorf("failed to update batch %d as checked, error: %v", batch.BatchNumber, err)
+		reprocessError(batch)
+		return nil, ErrUpdateBatchAsChecked
+	}
+
 	log.Infof("successful sanity check for batch %d, initialStateRoot: %s, stateRoot: %s, l2Blocks: %d, time: %v, used counters: %s",
 		batch.BatchNumber, initialStateRoot, batchResponse.NewStateRoot.String(), len(batchResponse.BlockResponses),
 		endProcessing.Sub(startProcessing), f.logZKCounters(batchResponse.UsedZkCounters))

sequencer/errors.go

@@ -14,6 +14,8 @@ var (
 	ErrReplacedTransaction = errors.New("replaced transaction")
 	// ErrGetBatchByNumber happens when we get an error trying to get a batch by number (GetBatchByNumber)
 	ErrGetBatchByNumber = errors.New("get batch by number error")
+	// ErrUpdateBatchAsChecked happens when we get an error trying to update a batch as checked (UpdateBatchAsChecked)
+	ErrUpdateBatchAsChecked = errors.New("update batch as checked error")
 	// ErrDecodeBatchL2Data happens when we get an error trying to decode BatchL2Data (DecodeTxs)
 	ErrDecodeBatchL2Data = errors.New("decoding BatchL2Data error")
 	// ErrProcessBatch happens when we get an error trying to process (executor) a batch