Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add slim endpoints for entities to speed up filters #460

Merged
merged 9 commits into from
Apr 19, 2020

Conversation

ghost
Copy link

@ghost ghost commented Apr 10, 2020

Should resolve issue #452

In my library with 123 performers and 1MB-ish performer images it speeds up AllPerformersForFilters from 0.4s to 0.04s. For 549 studios it speeds up from 0.08s to 0.02s.

I considered changing the dropdowns to fetch on demand, but considering that scenes can have arbitrary amount of performers attached, it would likely make page load significantly slower in many instances.

@ghost ghost changed the title Filter performance Add slim endpoints for entities to speed up filters Apr 10, 2020
Copy link
Collaborator

@bnkai bnkai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm 7x-8x speedup also for the performers and studios part
(536 Performers 627 Studios in my db)

@WithoutPants
Copy link
Collaborator

Testing against a database with 1,968 performers, I'm not seeing any difference before and after this change. I'm seeing between 7.5 and 12 seconds for allPerformersSlim to come back.

@ghost
Copy link
Author

ghost commented Apr 13, 2020

@WithoutPants Try now. I've added a migration that moves the image column to the end of the table. Apparently selecting columns past blobs incurs a large penalty in sqlite. On my instance it reduces the allPerformers query further down to 0.01s, though it seems my dataset isn't large enough to really stress the database at this point. I'm guessing at a certain size it spills to disk, causing the crazy query times.

@bnkai
Copy link
Collaborator

bnkai commented Apr 13, 2020

I was getting consistent speedup in the edit scene tab as stated above also. @WithoutPants is your db maybe in a spinning disk? ( my tests were in sata ssd and nvme ssd)

@bnkai
Copy link
Collaborator

bnkai commented Apr 13, 2020

With latest commit :
Database located on a nvme ssd
Edit Scene Tab
Firefox as browser

Time for AllPerformers
500+ performers
before 800ms after 15-20 ms
1k+ performers
before 1150ms after 25-30 ms

@WithoutPants
Copy link
Collaborator

I'm getting consistent times of around 50ms for allPerformersSlim now. Looks good.

Copy link
Collaborator

@WithoutPants WithoutPants left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should take the opportunity to remove the redundant index.

pkg/database/migrations/7_performer_optimization.up.sql Outdated Show resolved Hide resolved
@WithoutPants WithoutPants merged commit 2a3c974 into stashapp:develop Apr 19, 2020
Tweeticoats pushed a commit to Tweeticoats/stash that referenced this pull request Feb 1, 2021
* Move performers image column to end of table
* Remove redundant index
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Something needed tweaking.
Projects
None yet
2 participants