Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BCF-3244 Index only the fifth word to reduce the db size overhead #13315

Merged
merged 2 commits into from
May 24, 2024

Conversation

mateusz-sekara
Copy link
Collaborator

@mateusz-sekara mateusz-sekara commented May 24, 2024

Before the migration 0233

 schema_name | table_name |                  index_name                  | index_size
-------------+------------+----------------------------------------------+------------
 evm         | logs       | evm_logs_by_timestamp                        | 1274 MB
 evm         | logs       | idx_evm_logs_ordered_by_block_and_created_at | 1273 MB
 evm         | logs       | evm_logs_idx                                 | 1151 MB
 evm         | logs       | evm_logs_idx_tx_hash                         | 426 MB
 evm         | logs       | logs_pkey                                    | 272 MB
 evm         | logs       | evm_logs_idx_data_word_two                   | 167 MB
 evm         | logs       | evm_logs_idx_topic_three                     | 128 MB
 evm         | logs       | evm_logs_idx_data_word_three                 | 106 MB
 evm         | logs       | evm_logs_idx_data_word_one                   | 102 MB
 evm         | logs       | evm_logs_idx_topic_two                       | 92 MB
 evm         | logs       | evm_logs_idx_data_word_four                  | 91 MB
 evm         | logs       | evm_logs_idx_topic_four                      | 84 MB

 database_size
---------------
 12 GB

After the migration with only single index

 schema_name | table_name |                  index_name                  | index_size
-------------+------------+----------------------------------------------+------------
 evm         | logs       | evm_logs_by_timestamp                        | 1274 MB
 evm         | logs       | idx_evm_logs_ordered_by_block_and_created_at | 1273 MB
 evm         | logs       | evm_logs_idx                                 | 1151 MB
 evm         | logs       | evm_logs_idx_data_word_five                  | 1136 MB
 evm         | logs       | evm_logs_idx_tx_hash                         | 426 MB
 evm         | logs       | logs_pkey                                    | 272 MB
 evm         | logs       | evm_logs_idx_data_word_two                   | 167 MB
 evm         | logs       | evm_logs_idx_topic_three                     | 128 MB
 evm         | logs       | evm_logs_idx_data_word_three                 | 106 MB
 evm         | logs       | evm_logs_idx_data_word_one                   | 102 MB
 evm         | logs       | evm_logs_idx_topic_two                       | 92 MB
 evm         | logs       | evm_logs_idx_data_word_four                  | 91 MB
 evm         | logs       | evm_logs_idx_topic_four                      | 84 MB


 database_size
---------------
 13 GB

Applying the change directly to the migration script to prevent creating these indexes and then dropping them immediately - db size growing and then immediately shrinking

@mateusz-sekara mateusz-sekara changed the title Index only the fifth word to reduce the db size overhead BCF-3244 Index only the fifth word to reduce the db size overhead May 24, 2024
@reductionista reductionista marked this pull request as ready for review May 24, 2024 16:54
@reductionista reductionista requested a review from a team as a code owner May 24, 2024 16:54
@cl-sonarqube-production
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
84 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@reductionista reductionista added this pull request to the merge queue May 24, 2024
Merged via the queue into develop with commit 3af83ed May 24, 2024
107 checks passed
@reductionista reductionista deleted the only-fifth-word branch May 24, 2024 17:51
reductionista added a commit that referenced this pull request May 24, 2024
Co-authored-by: Domino Valdano <domino.valdano@smartcontract.com>
snehaagni added a commit that referenced this pull request May 25, 2024
…nly-fifth-word

BCF-3244 Index only the fifth word to reduce the db size overhead #13315
github-merge-queue bot pushed a commit that referenced this pull request Jun 6, 2024
* Bump version and update CHANGELOG for core v2.12.0

Signed-off-by: Sneha Agnihotri <sneha.agnihotri@smartcontract.com>

* core/services: fix ocrWrapper saveError contexts (#13139)

(cherry picked from commit 15fab1d)

* changed spammy error log to debug (#13153)

(cherry picked from commit 3502176)

* Updating changelog for 2.12.0  with cherry pick fix

Signed-off-by: Sneha Agnihotri <sneha.agnihotri@smartcontract.com>

* BCF-3225 - Implement forwarder fallback if forwarder not present as a transmitter on OCR2 aggregator (#13221)

* Implement forwarder OCR2 fallback if fwd not present as a transmitter

* Add changeset

* Fix panic on mercury server error (#13231) (#13256)

* Fix changelog

* Fixed CPU usage issues caused by inefficiencies in HeadTracker (#13230)

* Fixed CPU usage issues caused by inefficiencies in HeadTracker

* added comments

* revert heads back to the fix

(cherry picked from commit 6f1ebca)

* Update CHANGELOG for 2.12.0 bugfixes

Signed-off-by: Sneha Agnihotri <sneha.agnihotri@smartcontract.com>

* Decouple ChainType from config string [SHIP-2001] (#13272)

* fix: Decouple ChainType from config string

* fix: receiver name and failing test

* test: enhance config test to test for xdai specifically

* refactor: directly unmarshal into ChainType

* fix: validation

* test: fix TestDoc/EVM

* test: add xdai to warnings.xtar

* Index only the fifth word to reduce the db size overhead (#13315)

Co-authored-by: Domino Valdano <domino.valdano@smartcontract.com>

* Add special transmitter for OCR2 feeds (#13323)
* Add special transmitter for OCR2 feeds

* Add ocr2FeedsTransmitter FromAddress()

* Cherry-pick some forwarders context changes from (#13171)

* Improve ocr2FeedsTransmitter FromAddress() fallback (#13343)

* Improve ocr2FeedsTransmitter FromAddress() fallback

* Fix forwarders test error assert

* Update 2.12.0 Changelog with bugfixes

Signed-off-by: Sneha Agnihotri <sneha.agnihotri@smartcontract.com>

* core/services/pipeline: bridge task must continue after cancellation

* core/services/pipeline: use request context for deletion (#13404)

* core/services/pipeline: hide deadline from monitor

* Finalize date on changelog for 2.12.0

Signed-off-by: Sneha Agnihotri <sneha.agnihotri@smartcontract.com>

* Fix merge conflicts

* Remove `overtimeThresholds`

* Return to old version of pipeline-runner

---------

Signed-off-by: Sneha Agnihotri <sneha.agnihotri@smartcontract.com>
Co-authored-by: Jordan Krage <jmank88@gmail.com>
Co-authored-by: Patrick <patrick.huie@smartcontract.com>
Co-authored-by: ilija42 <57732589+ilija42@users.noreply.github.com>
Co-authored-by: Sam <samsondav@protonmail.com>
Co-authored-by: Dmytro Haidashenko <34754799+dhaidashenko@users.noreply.github.com>
Co-authored-by: Friedemann Fürst <friedemann.fuerst@smartcontract.com>
Co-authored-by: Mateusz Sekara <mateusz.sekara@smartcontract.com>
Co-authored-by: Domino Valdano <domino.valdano@smartcontract.com>
Co-authored-by: ilija <pavlovicilija42@gmail.com>
Co-authored-by: frank zhu <fr@nkzhu.com>
Co-authored-by: george-dorin <george.dorin@smartcontract.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants