Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LogPoller CLI command to resolve reorg greater than finality depth #12867

Merged
merged 11 commits into from
Apr 26, 2024

Conversation

dhaidashenko
Copy link
Collaborator

@dhaidashenko dhaidashenko commented Apr 17, 2024

Added new commands:

  • blocks find_lca - finds the latest block that is available in both the database and on the chain for the specified chain.
  • node remove-blocks removes all blocks and logs greater than or equal to the specified block number.

node remove-blocks is a local command and locks the database to ensure there is no data races within LogPoller or it's clients.

Copy link
Contributor

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

Copy link
Contributor

I see you added a changeset file but it does not contain a tag. Please edit the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@dhaidashenko dhaidashenko marked this pull request as ready for review April 19, 2024 16:11
@dhaidashenko dhaidashenko requested review from a team as code owners April 19, 2024 16:11
@@ -34,6 +36,18 @@ func initBlocksSubCmds(s *Shell) []cli.Command {
},
},
},
{
Name: "find_lca",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a convention for when to use _ vs - for clis? Not sure if there is but was curious for the reason for the difference in naming between find_lca and remove-blocks

"net/http"

"github.com/gin-gonic/gin"
"github.com/pkg/errors"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if possible can we use the standard errors package instead of pkg/errors. I think we can here. We are trying to reduce the use of this dependency since it is no longer maintained

---

Added a new CLI command, `blocks find_lca,` which finds the latest block that is available in both the database and on the chain for the specified chain.
Added a new CLI command, `node remove-blocks,` which removes all blocks and logs greater than or equal to the specified block number.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the rationale behind placing one of these commands under blocks and the other one under node. Seems like it might be more difficult to find them both if they're not in the same place?

Because these are both evm-specific commands, I think they should probably both go under chainlink chains evm... although I do notice there are some commands, such as chainlink blocks replay which are also evm-specific.
(We should probably move that one under chainlink chains evm soon also. I'll mention that to those more involved with LOOP arch planning.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it's confusing. I've just tried to follow existing structure of the commands.

Existing blocks replay interacts with the LogPoller and triggers it to replay blocks. find-lca - also interacts with the LogPoller and requests latest good block. Thus it made sense to me to group them together.
At the same time chains evm simply lists available evm chains.

Initially I've planed to have remove-blocks in the blocks section, but realised that it needs unique lock on the database. node/local sections seems to contain all commands that must be executed on local node only. It also contains rebroadcast-transactions, which is EVM/TXM related but still resides here. I figured that all local commands must be grouped together to highlight that they are executed on the local node.

Foundations are the owners of this package, so It's up to you to decide where to put the commands.
Should I move them?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, given that one of them is local and the other isn't I agree that it makes sense to have remove-blocks with rebroadcast-transactions and find-lca with replay. Unfortunately, I think we're going to want to move remove-blocks and rebroadcast-transactions soon to be under chainlink blocks evm and chainlink node evm. So I'm not sure whether it's better to add them now in the same place and move them later, or just add them in the right place now and move the other two later. Asking rest of my team for some advice on that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dhaidashenko Alright, yeah let's just add them here now and we can move them all together later.

@reductionista
Copy link
Contributor

I tried running this, and noticed that find_lca can only be run while the node is running, but remove_blocks can only be run when the node is stopped. I guess the first one makes sense, because you need a way to connect to the rpc servers, and without the node running there isn't one... is that right?

I wonder if the second one is too limiting though. I was thinking deleting recent blocks from evm.log_poller_blocks should be reasonably safe to do, even while LogPoller is running. But maybe there are some race conditions where it could cause funny behavior?

(Referring to db lease lock error message, when I try to run it while the node is running.)

@prashantkumar1982 prashantkumar1982 added this pull request to the merge queue Apr 26, 2024
Merged via the queue into develop with commit 27d9413 Apr 26, 2024
106 checks passed
@prashantkumar1982 prashantkumar1982 deleted the feature/BCI-2981-logpoller-resolve-reorg branch April 26, 2024 14:45
HenryNguyen5 added a commit that referenced this pull request Apr 27, 2024
DeividasK pushed a commit that referenced this pull request Apr 29, 2024
…12867)

* find lca and remove block after CLI

* fix sort.Find typo

* make RemoveBlocks local cmd

* tests

* added changeset

* added tags to the changeset

* fixed tests

* make cmds, vars cases consistent
github-merge-queue bot pushed a commit that referenced this pull request Apr 29, 2024
* implement update node operators in capability registry

* address PR feedback

* cache owner call

* Validate config on NewApplication (#12997)

* validate config on new application spawn

* Add changeset

* Check config only in the rebroadcast-transactions command

* Configurable Mercury transmitter parameters (#12680)

* Configurable Mercury transmitter parameters

* Changeset

* Remove commented code

* add tag

* Rename

* LogPoller CLI command to resolve reorg greater than finality depth (#12867)

* find lca and remove block after CLI

* fix sort.Find typo

* make RemoveBlocks local cmd

* tests

* added changeset

* added tags to the changeset

* fixed tests

* make cmds, vars cases consistent

* Fix Node Migration Test Check For Versions (#12982)

* fix: prevent query syntax error if allowlist is empty (#12912)

Co-authored-by: Morgan Kuphal <87319522+KuphJr@users.noreply.github.com>

* Update wrappers

* Formatting

---------

Co-authored-by: george-dorin <120329946+george-dorin@users.noreply.github.com>
Co-authored-by: Sam <samsondav@protonmail.com>
Co-authored-by: Dmytro Haidashenko <34754799+dhaidashenko@users.noreply.github.com>
Co-authored-by: Tate <tate.exon@smartcontract.com>
Co-authored-by: Gabriel Paradiso <gabriel.paradiso@smartcontract.com>
Co-authored-by: Morgan Kuphal <87319522+KuphJr@users.noreply.github.com>
Co-authored-by: DeividasK <deividas.karzinauskas@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants