Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move image blobs into separate tables #618

Merged
merged 10 commits into from
Jun 22, 2020

Conversation

WithoutPants
Copy link
Collaborator

Moves all images into separate tables:

  • scene cover into scenes_cover
  • performer image into performers_image
  • studio image into studios_image
  • movie front_image/back_image into movies_images

The performance gain is potentially quite large. On a database with ~1000 scenes, all with covers, the following graphql query went from 6.39 seconds to 477ms!

graphql query in question (tested using graphql playground):

{
  findScenes(filter: {
    page: 1,
    per_page:500,
    sort: "date",
    direction:ASC
  }) {
    count
    scenes {
      id
  checksum
  title
  details
  url
  date
  rating
  o_counter
  path

  file {
    size
    duration
    video_codec
    audio_codec
    width
    height
    framerate
    bitrate
  }

  paths {
    screenshot
    preview
    stream
    webp
    vtt
    chapters_vtt
  }

  scene_markers {
    id
    title
    seconds
  }

  gallery {
    id
    path
    title
  }

  studio {
    id
    name
    image_path
  }

  movies {
    movie {
      id
      name
      front_image_path
    }
    scene_index
  }

  tags {
    id
    name
  }

  performers {
    id
    name
    favorite
    image_path
  }
    }
  }
}

This required a schema migration that essentially rebuilds the entire schema, since removing columns requires rebuilding both the table and referencing tables. As such, I have performed the following verification steps to ensure there is no loss of data:

  • ran .schema on the before and after databases and compared the results to ensure no fields or indexes were lost
  • ran an export on the before and after versions and ensured that they were identical
  • ran an import of the exported after data, then re-exported and ensured that the new export was identical to the previous one

The only part that hasn't been exercised by this is movies, since the database I was using didn't have them. I will test that in the meantime.

Other changes:

  • added utils.ServeImage which performs the etag generation added previous and applies to all database images
  • scene cover will fallback to use the database cover image if the generated image is not found
  • perform a vacuum on the database after all migrations are completed
  • add logging to the migration process
  • change Box.Bytes to Box.Find to eliminate the deprecation message

Other things to consider:

  • performer, studio and movie images are no longer mandatory in the schema. This means we can look at not generating these images at creation, but serve the correct "random" image based on the checksum of the object if the database image is null. This will also allow querying for performers/studios with no image
  • the movie images table should be changed so that one of the images is mandatory, not just front_image

Resolves #425

@WithoutPants WithoutPants added the improvement Something needed tweaking. label Jun 18, 2020
@WithoutPants WithoutPants added this to the Version 0.3.0 milestone Jun 18, 2020
@WithoutPants
Copy link
Collaborator Author

Thanks for the heads up. I've since tested against some movies and fixed an unrelated issue that movie studios were not being exported or imported.

@WithoutPants WithoutPants merged commit 7a74658 into stashapp:develop Jun 22, 2020
WithoutPants added a commit that referenced this pull request Jun 22, 2020
Tweeticoats pushed a commit to Tweeticoats/stash that referenced this pull request Feb 1, 2021
* Scene cover fallback to database
* Fix panic if studio not found
* Fix movie studio not being imported/exported
Tweeticoats pushed a commit to Tweeticoats/stash that referenced this pull request Feb 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Something needed tweaking.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[RFC] Changing image storage from DB blob to file system
2 participants