Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: background worker routines to shutdown client for migration #2538

Merged
merged 26 commits into from
Aug 6, 2024

Conversation

kingpinXD
Copy link
Contributor

@kingpinXD kingpinXD commented Jul 24, 2024

Description

This pr adds three background threads which can trigger the zeta-client to shutdown to help the process of migration
Thread 1 : Check for keygen and restart if new keygen is set
Thread 2 : Check TSS list and restart if a new tss is added
Thread 3 : Check current TSS , and restart if it is updated

Closes #2455
#2492

How Has This Been Tested?

  • Tested CCTX in localnet
  • Tested in development environment
  • Go unit tests
  • Go integration tests
  • Tested via GitHub Actions

Summary by CodeRabbit

  • New Features

    • Introduced structured background processing for handling TSS updates and key generation.
    • Added a new Logger() method to access logging capabilities within the application context.
    • Implemented a new method GetTSS to retrieve TSS objects directly from the observer client.
    • Added a constant backoff strategy for retry mechanisms.
  • Improvements

    • Enhanced gRPC connection security.
    • Improved error handling and logging during shutdown processes.
    • Updated restart policies in Docker configuration to ensure service resilience.
  • Bug Fixes

    • Addressed issues with task cancellation and shutdown sequence management.

Copy link
Contributor

coderabbitai bot commented Jul 24, 2024

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

Walkthrough

The recent changes enhance the zetaclient module's management of TSS (Threshold Signature Scheme) updates and key generation, shifting responsibilities for restart logic directly into the client. The refactoring improves error handling and logging, introduces robust background processing, and strengthens gRPC security. Additionally, Docker Compose files now ensure automatic service restarts, further increasing application resilience.

Changes

Files Change Summary
cmd/zetaclientd-supervisor/lib.go, cmd/zetaclientd-supervisor/main.go, cmd/zetaclientd/start.go Removed obsolete TSS functions and improved logging/error handling; enhanced background processing for TSS operations.
contrib/localnet/docker-compose.yml Added restart: always directive to services for automatic restarts on failure.
pkg/bg/bg.go Introduced callback field in config struct and WithCallback functional option for task management.
pkg/retry/retry.go Added DefaultConstantBackoff function for consistent retry intervals.
zetaclient/context/app.go New Logger() method in AppContext for enhanced logging capabilities.
zetaclient/zetacore/background_threads.go Implemented three new background handlers for monitoring TSS and key generation.
zetaclient/zetacore/client_query_observer.go Added GetTSS method to retrieve TSS objects, enhancing client functionality.
zetaclient/orchestrator/orchestrator.go Simplified shutdown process in MonitorCore by removing explicit observer stopping logic.

Sequence Diagram(s)

sequenceDiagram
    participant Main
    participant Supervisor
    participant BackgroundThreads
    participant Client

    Main->>Supervisor: Start supervisor
    Supervisor->>BackgroundThreads: Initiate TSS handling
    BackgroundThreads->>Client: Monitor TSS updates
    Client-->>BackgroundThreads: Notify TSS change
    BackgroundThreads->>Supervisor: Log update and exit
    Supervisor-->>Main: Notify completion
Loading

Assessment against linked issues

Objective Addressed Explanation
Move restart logic from supervisor into the zetaclient directly (#2455)
Ensure proper handling of TSS updates and key generation restarts (#2455)

🐇 In the code where changes flow,
The TSS updates now surely grow.
With logs that shine and errors that fade,
Our little app is better made!
Restarting services, oh what a treat,
Hopping along, it can’t be beat! 🌟


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@kingpinXD kingpinXD added the TSS_MIGRATION_TESTS Run TSS migration tests label Jul 24, 2024
@kingpinXD kingpinXD marked this pull request as ready for review July 24, 2024 01:29
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Outside diff range, codebase verification and nitpick comments (1)
zetaclient/orchestrator/orchestrator.go (1)

Line range hint 65-71:
Ensure proper error handling for OnBeforeStop.

The OnBeforeStop method of oc.zetacoreClient should have proper error handling to ensure that any issues during the shutdown process are logged and managed appropriately.

+  if err := oc.zetacoreClient.OnBeforeStop(shutdownOrchestrator); err != nil {
+    oc.logger.Std.Error().Err(err).Msg("error during OnBeforeStop")
+  }
Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between e6ce681 and 731b03e.

Files selected for processing (10)
  • cmd/zetaclientd-supervisor/lib.go (5 hunks)
  • cmd/zetaclientd-supervisor/main.go (1 hunks)
  • cmd/zetaclientd/start.go (6 hunks)
  • contrib/localnet/docker-compose.yml (2 hunks)
  • pkg/bg/bg.go (4 hunks)
  • pkg/retry/retry.go (1 hunks)
  • zetaclient/context/app.go (1 hunks)
  • zetaclient/orchestrator/orchestrator.go (1 hunks)
  • zetaclient/zetacore/background_threads.go (1 hunks)
  • zetaclient/zetacore/client_query_observer.go (1 hunks)
Additional context used
Path-based instructions (9)
pkg/bg/bg.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

cmd/zetaclientd-supervisor/main.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

pkg/retry/retry.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/zetacore/background_threads.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/zetacore/client_query_observer.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

cmd/zetaclientd-supervisor/lib.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/context/app.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

cmd/zetaclientd/start.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/orchestrator/orchestrator.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

Additional comments not posted (19)
pkg/bg/bg.go (3)

14-14: LGTM! The new cancel field enhances flexibility.

Adding cancel of type context.CancelCauseFunc allows for more robust task management.


23-25: LGTM! The new WithCancel function adds flexibility.

The WithCancel function allows users to set the cancel field, enhancing the configuration's flexibility.


Line range hint 36-60: LGTM! The Work function changes enhance functionality.

The Work function now uses the cancel function if provided, allowing the background task to signal a restart based on its outcome.

cmd/zetaclientd-supervisor/main.go (1)

72-81: LGTM! Enhanced error handling and logging.

The changes improve the visibility of the process's exit status and manage the lifecycle of goroutines more effectively.

pkg/retry/retry.go (1)

49-52: LGTM! The new DefaultConstantBackoff function enhances retry capabilities.

The DefaultConstantBackoff function provides a specific backoff configuration, enhancing the retry capabilities.

zetaclient/zetacore/background_threads.go (3)

14-59: LGTM! Ensure the ticker interval is appropriate.

The function is well-structured with proper error handling and logging. The use of retry logic for initial TSS retrieval is a good practice. Verify that the ticker interval of 5 seconds is appropriate for your use case.


61-110: LGTM! Ensure the ticker interval is appropriate.

The function is well-structured with proper error handling and logging. The use of retry logic for initial TSS history retrieval is a good practice. Verify that the ticker interval of 5 seconds is appropriate for your use case.


112-161: LGTM! Ensure the ticker interval is appropriate.

The function is well-structured with proper error handling and logging. The use of retry logic for initial keygen retrieval is a good practice. Verify that the ticker interval of 5 seconds is appropriate for your use case.

zetaclient/zetacore/client_query_observer.go (1)

170-176: LGTM! Ensure the retry logic is correctly implemented.

The method is well-structured with proper error handling. The use of retry logic is a good practice. Verify that the retry logic is correctly implemented.

cmd/zetaclientd-supervisor/lib.go (2)

22-22: LGTM! Ensure the new method is correctly implemented.

The change improves security by using grpc.WithTransportCredentials(insecure.NewCredentials()) instead of the deprecated grpc.WithInsecure(). Verify that the new method is correctly implemented.

Also applies to: 81-81


225-226: LGTM! Ensure consistent use of the new variable name.

The change improves code readability by changing the variable name config to cfg. Verify that the new variable name is consistently used throughout the code.

Also applies to: 234-234

contrib/localnet/docker-compose.yml (1)

127-127: Ensure service resilience with restart: always.

The addition of restart: always ensures that the services automatically restart if they stop or crash, enhancing resilience and uptime.

Also applies to: 144-144

zetaclient/context/app.go (1)

73-75: New method Logger() added to AppContext.

The new method provides direct access to the logger instance, which enhances the functionality by allowing logging capabilities through the context object.

cmd/zetaclientd/start.go (5)

216-218: Create a notification context for background threads.

The notifyCtx context is created using context.WithCancelCause, allowing coordinated shutdown signals between the main thread and background processes.


220-220: Start background threads.

The startBackgroundThreads function is deferred to ensure background threads are started and managed properly.


330-338: Clarify variable naming for orchestrator.

The variable name change from orchestrator to cctxOrchestrator enhances clarity in the code.


362-370: Refine shutdown signal handling.

The shutdown signal handling is refined using a select statement to wait for either the notifyCtx to be done or a system interrupt signal, improving robustness.


439-471: Introduce startBackgroundThreads function.

The startBackgroundThreads function initiates multiple background tasks, each responsible for handling specific TSS-related operations, enhancing the maintainability and functionality of the code.

zetaclient/orchestrator/orchestrator.go (1)

Line range hint 65-71:
Verify the correctness of the new shutdown process.

The shutdown process now relies on closing the oc.stop channel and invoking the OnBeforeStop method of oc.zetacoreClient. Ensure that this new approach correctly stops all observers and performs any necessary cleanup.

Verification successful

The new shutdown process correctly stops all observers and performs necessary cleanup.

The oc.stop channel is closed to signal the shutdown, and the OnBeforeStop method is used to perform any necessary cleanup before stopping.

  • Closing the oc.stop channel correctly stops the orchestrator.
  • The OnBeforeStop method ensures necessary cleanup is performed before shutdown.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify that the new shutdown process correctly stops all observers and performs necessary cleanup.

# Test: Search for the `Stop` method in the codebase. Expect: Ensure that the `Stop` method is not required for proper shutdown.
rg --type go -A 5 $'func Stop'

Length of output: 31


Script:

#!/bin/bash
# Description: Verify the usage of the `oc.stop` channel and `OnBeforeStop` method in the codebase.

# Search for the usage of the `oc.stop` channel.
rg --type go 'oc\.stop'

# Search for the `OnBeforeStop` method in the codebase.
rg --type go 'OnBeforeStop'

# Verify if there are any other shutdown or cleanup mechanisms in place.
rg --type go -A 5 'func Shutdown'

Length of output: 770


Script:

#!/bin/bash
# Description: Review the implementation of the `OnBeforeStop` method and the usage of the `oc.stop` channel.

# Review the implementation of the `OnBeforeStop` method.
rg --type go -A 10 'func (c *Client) OnBeforeStop' zetaclient/zetacore/client.go

# Review the usage of the `oc.stop` channel in the orchestrator code.
rg --type go -A 10 'oc\.stop' zetaclient/orchestrator/orchestrator.go

Length of output: 699

cmd/zetaclientd-supervisor/main.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
Copy link
Member

@lumtis lumtis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to have a review from @swift1337 for this PR. 🙏

cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
cmd/zetaclientd-supervisor/main.go Outdated Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Show resolved Hide resolved
zetaclient/context/app.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 731b03e and 2cce32c.

Files selected for processing (4)
  • cmd/zetaclientd-supervisor/main.go (1 hunks)
  • cmd/zetaclientd/start.go (3 hunks)
  • pkg/bg/bg.go (4 hunks)
  • zetaclient/zetacore/client_tss_migration_listners.go (1 hunks)
Files skipped from review as they are similar to previous changes (2)
  • cmd/zetaclientd-supervisor/main.go
  • cmd/zetaclientd/start.go
Additional context used
Path-based instructions (2)
pkg/bg/bg.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/zetacore/client_tss_migration_listners.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

Additional comments not posted (8)
pkg/bg/bg.go (4)

14-14: LGTM!

The addition of the cancel field enhances the flexibility of the configuration.


23-25: LGTM!

The WithCancel function allows users to set the cancel field when configuring the config struct, enhancing the flexibility of the configuration.


36-36: LGTM!

The Work function initializes the cancel field to nil.


51-59: LGTM!

The Work function now includes logic to use the cancel function if it is provided, allowing the background task to signal a cancellation based on its outcome. This enhances error handling and control flow.

zetaclient/zetacore/client_tss_migration_listners.go (4)

18-45: LGTM!

The StartTssMigrationRoutines function starts background threads for handling TSS updates, new keygen, and new TSS key generation. The use of the WithCancel option allows the main thread to be restarted based on the outcome of the background tasks.


48-92: LGTM!

The HandleTSSUpdate function listens for TSS updates and returns when the TSS address is updated. The function handles errors and logs warnings if it fails to retrieve the TSS.


94-142: LGTM!

The HandleNewTSSKeyGeneration function listens for new TSS key generation and returns when a new key is generated. The function handles errors and logs warnings if it fails to retrieve the TSS history.


144-193: LGTM!

The HandleNewKeygen function listens for new keygen and returns when a new keygen is set. The function handles errors and logs warnings if it fails to retrieve the keygen.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 2cce32c and cae80f2.

Files selected for processing (3)
  • cmd/zetaclientd-supervisor/main.go (2 hunks)
  • pkg/bg/bg.go (3 hunks)
  • zetaclient/zetacore/client_tss_migration_listners.go (1 hunks)
Files skipped from review as they are similar to previous changes (2)
  • cmd/zetaclientd-supervisor/main.go
  • zetaclient/zetacore/client_tss_migration_listners.go
Additional context used
Path-based instructions (1)
pkg/bg/bg.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

Additional comments not posted (3)
pkg/bg/bg.go (3)

12-14: LGTM!

The addition of the callback field in the config struct enhances flexibility by allowing for a cancellation mechanism.


23-25: LGTM!

The new function WithCallback provides a useful way to set the callback field in the config struct.


34-36: Suggestion: Initialize callback field to nil.

The initialization of the callback field to nil is correct and ensures that the callback mechanism is optional.

pkg/bg/bg.go Outdated Show resolved Hide resolved
@kingpinXD kingpinXD linked an issue Jul 29, 2024 that may be closed by this pull request
changelog.md Outdated Show resolved Hide resolved
zetaclient/zetacore/client_query_observer.go Show resolved Hide resolved
zetaclient/zetacore/client_query_observer.go Show resolved Hide resolved
@lumtis lumtis self-requested a review July 30, 2024 10:48
@lumtis
Copy link
Member

lumtis commented Jul 30, 2024

Let's wait on #2560 before considering merging this one

pkg/bg/bg.go Outdated Show resolved Hide resolved
zetaclient/zetacore/client_tss_migration_listners.go Outdated Show resolved Hide resolved
zetaclient/zetacore/background_threads.go Outdated Show resolved Hide resolved
Copy link
Contributor

@swift1337 swift1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@lumtis lumtis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code organization and implementation looks good to me, I guess this can be tested E2E in the TSS migration test?

@kingpinXD
Copy link
Contributor Author

Code organization and implementation looks good to me, I guess this can be tested E2E in the TSS migration test?

Yes , the TSS migration test would use these routines

@lumtis lumtis changed the title feat: background threads to shutdown client for migration feat: background worker routines to shutdown client for migration Aug 5, 2024
@kingpinXD kingpinXD added this pull request to the merge queue Aug 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 6, 2024
@kingpinXD kingpinXD added this pull request to the merge queue Aug 6, 2024
Merged via the queue into develop with commit d94047a Aug 6, 2024
27 of 28 checks passed
@kingpinXD kingpinXD deleted the restart-thread-zetaclient branch August 6, 2024 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use retry package for queries on zetaclient Move restart logic from supervisor into the zetaclient directly
5 participants