Db stuffz #3

stefanpenner · 2014-09-21T03:16:38Z

No description provided.

alexcrichton · 2014-09-21T03:20:37Z

src/app.rs

@@ -57,6 +57,7 @@ impl App {
    }

    pub fn db_setup(&self) {
+        print!(" - setting up the db\n")


This is normally idiomatically written as:

println!(" - setting up the db");

Db stuffz

# This is the 1st commit message: /crates page, better mobile styling # The commit message rust-lang#2 will be skipped: # changed data-test attr for class # The commit message rust-lang#3 will be skipped: # bottom positioning of header element on crates page

Our database spends more of its time processing /api/v1/crates with no parameters other than pagination. This query is the main one hit by crawlers, and it is taking over 100ms to run, so it's at the top of our list (for posterity's sake, rust-lang#2 is copying `crate_downloads` during backups, rust-lang#3 and rust-lang#4 are the updates run from bin/update-downloads, and rust-lang#5 is the query run from the download endpoint) The query is having to perform the full join between crates and recent_downloads, and then count the results of that. Since we have no search parameters of any kind, this count is equivalent to just counting the crates table, which we can do much more quickly. We still need to do the count over the whole thing if there's any where clause, but we can optimize the case where there's no search. This implicitly relies on the fact that we're only changing the select clause in branches where we're also setting a where clause. Diesel 2 will probably have a feature that lets us avoid this. We could also refactor the "exact match" check to be client side instead of the DB and get rid of all the cases where we modify the select clause. Before: ``` Limit (cost=427.87..470.65 rows=100 width=877) (actual time=109.698..109.739 rows=100 loops=1) -> WindowAgg (cost=0.14..10119.91 rows=23659 width=877) (actual time=109.277..109.697 rows=1100 loops=1) -> Nested Loop Left Join (cost=0.14..9966.13 rows=23659 width=869) (actual time=0.051..85.429 rows=23659 loops=1) -> Index Scan using index_crates_name_ordering on crates (cost=0.08..7604.30 rows=23659 width=860) (actual time=0.037..34.975 rows=23659 loops=1) -> Index Scan using recent_crate_downloads_crate_id on recent_crate_downloads (cost=0.06..0.10 rows=1 width=12) (actual time=0.002..0.002 rows=1 loops=23659) Index Cond: (crate_id = crates.id) Planning time: 1.307 ms Execution time: 111.840 ms ``` After: ``` Limit (cost=1052.34..1094.76 rows=100 width=877) (actual time=11.536..12.026 rows=100 loops=1) InitPlan 1 (returns $0) -> Aggregate (cost=627.96..627.96 rows=1 width=8) (actual time=4.966..4.966 rows=1 loops=1) -> Index Only Scan using packages_pkey on crates crates_1 (cost=0.06..616.13 rows=23659 width=0) (actual time=0.015..3.513 rows=23659 loops=1) Heap Fetches: 811 -> Subquery Scan on t (cost=0.14..10037.11 rows=23659 width=877) (actual time=5.019..11.968 rows=1100 loops=1) -> Nested Loop Left Join (cost=0.14..9966.13 rows=23659 width=869) (actual time=0.051..6.831 rows=1100 loops=1) -> Index Scan using index_crates_name_ordering on crates (cost=0.08..7604.30 rows=23659 width=860) (actual time=0.038..3.331 rows=1100 loops=1) -> Index Scan using recent_crate_downloads_crate_id on recent_crate_downloads (cost=0.06..0.10 rows=1 width=12) (actual time=0.003..0.003 rows=1 loops=1100) Index Cond: (crate_id = crates.id) Planning time: 1.377 ms Execution time: 12.106 ms ```

Optimize our most time consuming query Our database spends more of its time processing /api/v1/crates with no parameters other than pagination. This query is the main one hit by crawlers, and it is taking over 100ms to run, so it's at the top of our list (for posterity's sake, #2 is copying `crate_downloads` during backups, #3 and #4 are the updates run from bin/update-downloads, and #5 is the query run from the download endpoint) The query is having to perform the full join between crates and recent_downloads, and then count the results of that. Since we have no search parameters of any kind, this count is equivalent to just counting the crates table, which we can do much more quickly. We still need to do the count over the whole thing if there's any where clause, but we can optimize the case where there's no search. This implicitly relies on the fact that we're only changing the select clause in branches where we're also setting a where clause. Diesel 2 will probably have a feature that lets us avoid this. We could also refactor the "exact match" check to be client side instead of the DB and get rid of all the cases where we modify the select clause. Before: ``` Limit (cost=427.87..470.65 rows=100 width=877) (actual time=109.698..109.739 rows=100 loops=1) -> WindowAgg (cost=0.14..10119.91 rows=23659 width=877) (actual time=109.277..109.697 rows=1100 loops=1) -> Nested Loop Left Join (cost=0.14..9966.13 rows=23659 width=869) (actual time=0.051..85.429 rows=23659 loops=1) -> Index Scan using index_crates_name_ordering on crates (cost=0.08..7604.30 rows=23659 width=860) (actual time=0.037..34.975 rows=23659 loops=1) -> Index Scan using recent_crate_downloads_crate_id on recent_crate_downloads (cost=0.06..0.10 rows=1 width=12) (actual time=0.002..0.002 rows=1 loops=23659) Index Cond: (crate_id = crates.id) Planning time: 1.307 ms Execution time: 111.840 ms ``` After: ``` Limit (cost=1052.34..1094.76 rows=100 width=877) (actual time=11.536..12.026 rows=100 loops=1) InitPlan 1 (returns $0) -> Aggregate (cost=627.96..627.96 rows=1 width=8) (actual time=4.966..4.966 rows=1 loops=1) -> Index Only Scan using packages_pkey on crates crates_1 (cost=0.06..616.13 rows=23659 width=0) (actual time=0.015..3.513 rows=23659 loops=1) Heap Fetches: 811 -> Subquery Scan on t (cost=0.14..10037.11 rows=23659 width=877) (actual time=5.019..11.968 rows=1100 loops=1) -> Nested Loop Left Join (cost=0.14..9966.13 rows=23659 width=869) (actual time=0.051..6.831 rows=1100 loops=1) -> Index Scan using index_crates_name_ordering on crates (cost=0.08..7604.30 rows=23659 width=860) (actual time=0.038..3.331 rows=1100 loops=1) -> Index Scan using recent_crate_downloads_crate_id on recent_crate_downloads (cost=0.06..0.10 rows=1 width=12) (actual time=0.003..0.003 rows=1 loops=1100) Index Cond: (crate_id = crates.id) Planning time: 1.377 ms Execution time: 12.106 ms ```

Switch from `futures-cpupool` to `tokio-threadpool`

Replace HashMap<&'static str, Box<Any>> with TypeMap.

Update dependencies and version

update deps and version

Add dyn keyword to address warnings

Bump dependencies for new alpha release

Fixes #3

little feedback when the DB is being setup

f6c5b0b

alexcrichton reviewed Sep 21, 2014
View reviewed changes

add a note for setting up the database

a2c4720

stefanpenner force-pushed the db branch from 0b620ec to a2c4720 Compare September 21, 2014 03:25

alexcrichton added a commit that referenced this pull request Sep 21, 2014

Merge pull request #3 from stefanpenner/db

cb135a3

Db stuffz

alexcrichton merged commit cb135a3 into rust-lang:master Sep 21, 2014

stefanpenner deleted the db branch September 21, 2014 03:28

tarcieri mentioned this pull request Dec 12, 2015

Security model / TUF #75

Open

sgrif mentioned this pull request Mar 14, 2019

Optimize our most time consuming query #1668

Merged

Turbo87 pushed a commit that referenced this pull request Dec 8, 2022

Merge pull request #3 from jtgeibel/tokio-threadpool

a745cf5

Switch from `futures-cpupool` to `tokio-threadpool`

Turbo87 pushed a commit that referenced this pull request Jan 4, 2023

Merge pull request #3 from reem/typemap

fb1a332

Replace HashMap<&'static str, Box<Any>> with TypeMap.

Turbo87 pushed a commit that referenced this pull request Jan 4, 2023

Merge pull request #3 from steveklabnik/master

a292938

Update dependencies and version

Turbo87 pushed a commit that referenced this pull request Jan 4, 2023

Merge pull request #3 from steveklabnik/master

78553d3

update deps and version

Turbo87 pushed a commit that referenced this pull request Jan 4, 2023

Merge pull request #3 from jtgeibel/fix-warnings

a5d7ab7

Add dyn keyword to address warnings

Turbo87 pushed a commit that referenced this pull request Jan 4, 2023

Merge pull request #3 from jtgeibel/bump-versions

272a803

Bump dependencies for new alpha release

Turbo87 pushed a commit that referenced this pull request Jan 4, 2023

Upgrade to time 0.2

6805f64

Fixes #3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Db stuffz #3

Db stuffz #3

stefanpenner commented Sep 21, 2014

alexcrichton Sep 21, 2014

Db stuffz #3

Db stuffz #3

Conversation

stefanpenner commented Sep 21, 2014

alexcrichton Sep 21, 2014

Choose a reason for hiding this comment