Commits on Apr 23, 2024

1. Adding evm_chain_id, address and event_sig to data and topic indexes … …

   …(#12786)
   
   * Adding evm_chain_id, address and event_sig to data and topic indexes
   
   * Post review fixes
   
   * Post review fixes
   
   * Post review fixes
   
   * Post review fixes
   
   * Bumping migration script number
   
   ---------
   
   Co-authored-by: Domino Valdano <domino.valdano@smartcontract.com>

   

   mateusz-sekara and reductionista committed Apr 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 5fd24d6
   • Browse repository at this point

   Copy the full SHA

   5fd24d6 View commit details

   Browse the repository at this point in the history

2. Improved fetching Commit Reports from database (#726) …

   Current implementation fetches all the CommitReports within the
   permissionlessExecThreshold from database and filter out result set
   based on the `snoozedRoots`. It means that when everything is executed,
   we keep fetching all roots to memory only to discover that we don't need
   that data. During high traffic spikes, this is an unnecessary waste of
   resources in the CL node and database as the dataset size grows in time
   till it reaches permissionlessExec window (8 hours of logs for prod).
   
   Example load run in which you can see how it changes over time during
   steady traffic of messages
   
   <img width="1488" alt="image"
   src="https://github.com/smartcontractkit/ccip/assets/18304098/7b216dd4-1ab3-4943-9797-99a38f4f9eaa">
   
   There are at least two solutions to make it more performant
   
   1. Pass executed roots to the database query to fetch only unexpired
   roots from the database. Unexpired roots can be determined from the
   snoozedRoots cache. This is probably the most accurate approach because
   it always fetches the minimal required set from the database.
   
   You can think of the query like this
   ```sql
   select * from evm.logs
   where address = :addr
   and event_sig = :event_sig
   and substring(data, X, Y) not in (:executed_roots_array)
   and block_timestamp > :permissionlessThresholdWindow
   ```
   
   2. Make permissionlessThreshold filter "sliding". Commit Reports are
   executed sequentially for most of the cases, so we can keep track of
   timestamps of the oldest fully executed report. Instead of inventing a
   new query for LogPoller we can achieve a lot based on the assumption
   that execution is sequential. Whenever report is markedAsExecuted we
   keep bumping block_timestamp filter. We already have that information in
   cache and indirectly rely on that when filtering out snoozedRoots
   ```sql
   and block_timestamp > :oldest_executed_timestamp
   ```
   
   Solution 2 is easier to implement, doesn't require any changes to
   LogPoller (and query perf testing) and can be handled in the plugin. We
   already cache snoozedRoots so we can reuse that to move filtering to the
   database by narrowing the block_timestamp search window. Assuming that
   reports are executed sequentially, it should be as accurate as solution
   1.
   
   duration)
   <img width="1088" alt="image"
   src="https://github.com/smartcontractkit/ccip/assets/18304098/c445495d-51e7-4138-a00b-662423fa0cdf">

   

   mateusz-sekara committed Apr 23, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 8c82c7a
   • Browse repository at this point

   Copy the full SHA

   8c82c7a View commit details

   Browse the repository at this point in the history