Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Studio Tagger #3510

Merged
merged 42 commits into from
Jul 30, 2023
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
c7266b9
Studio image and parent studio support in scene tagger
Flashy78 Feb 19, 2023
e8ab7da
Merge branch 'develop' into studio-image
Flashy78 Feb 19, 2023
0cdd331
Refactor studio backend and add studio tagger
Flashy78 Feb 26, 2023
784656c
Refactor studio backend and add studio tagger
Flashy78 Feb 26, 2023
2c8eeca
Merge branch 'studio-image' of github.com:Flashy78/stash into studio-…
Flashy78 Mar 7, 2023
49e4621
Merge branch 'develop' into studio-image
Flashy78 Mar 7, 2023
599415c
Fix merge
Flashy78 Mar 7, 2023
ce735e0
Removed ScrapedParentStudio, added performerids/names, revert timesta…
Flashy78 Mar 8, 2023
6b783fa
Fix possible missing value
Flashy78 Mar 14, 2023
feddb19
Merge branch 'develop' into studio-image
Flashy78 Apr 12, 2023
22c880a
Fix merge
Flashy78 Apr 18, 2023
d11bd6e
Format and tests
Flashy78 Apr 18, 2023
476f5f6
Remove linter fix
Flashy78 Apr 18, 2023
fa51ae8
Merge branch 'develop' into studio-image
Flashy78 May 23, 2023
da99a6c
Merge branch 'develop' into studio-image
Flashy78 Jun 16, 2023
65d1790
Refactoring from merge
Flashy78 Jun 16, 2023
60db199
Fix updatedAt duplicate
Flashy78 Jun 16, 2023
20eb282
Add internal find method back
Flashy78 Jun 20, 2023
5b79ae5
Moved parent studio handling out of sqlite package
Flashy78 Jun 23, 2023
43279c5
Revert unneeded changes
Flashy78 Jun 23, 2023
a2884cf
Adding error checking
Flashy78 Jun 23, 2023
cfc0144
Added error check
Flashy78 Jun 23, 2023
20b66c2
Merge remote-tracking branch 'upstream/develop' into prs/3510
WithoutPants Jul 12, 2023
a49b44e
Remove parent create/update field
WithoutPants Jul 13, 2023
801e076
Code style
WithoutPants Jul 13, 2023
a5d4511
Refactor studio partial image field
WithoutPants Jul 13, 2023
bc50822
Bug fixes
WithoutPants Jul 13, 2023
44ac1e1
Merge remote-tracking branch 'upstream/develop' into prs/3510
WithoutPants Jul 13, 2023
fdc92f8
Set created/updated times
WithoutPants Jul 13, 2023
7881b2c
Revert unnecessary studio sqlite changes
WithoutPants Jul 13, 2023
cf68d39
Move scraped studio test
WithoutPants Jul 13, 2023
daf4dcf
Use read txn for relationship resolvers
WithoutPants Jul 13, 2023
7f2b84e
Remove image data from studio and partial
WithoutPants Jul 13, 2023
ef5f667
Code style
WithoutPants Jul 13, 2023
8c33574
Fix graphql error when opening studio modal
WithoutPants Jul 13, 2023
c367c13
Localisation
WithoutPants Jul 13, 2023
2a8bee4
Refactor studio modal
WithoutPants Jul 14, 2023
d224c34
Handle parent studios in taggers
WithoutPants Jul 14, 2023
4c84a93
Merge remote-tracking branch 'upstream/develop' into prs/3510
WithoutPants Jul 27, 2023
741ce2f
Fix saving when parent exists
WithoutPants Jul 27, 2023
86b4031
Merge remote-tracking branch 'upstream/develop' into prs/3510
WithoutPants Jul 28, 2023
b10c8e9
Always refetch stash-box performer/studio
WithoutPants Jul 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ NOTE: The `make` command in Windows will be `mingw32-make` with MingW. For examp

* `make pre-ui` - Installs the UI dependencies. Only needs to be run once before building the UI for the first time, or if the dependencies are updated
* `make generate` - Generate Go and UI GraphQL files
* `make generate-stash-box-client` - Generate Go files from a Stash-box
* `make fmt-ui` - Formats the UI source code
* `make ui` - Builds the frontend
* `make build` - Builds the binary (make sure to build the UI as well... see below)
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ require (
github.com/disintegration/imaging v1.6.0
github.com/fvbommel/sortorder v1.0.2
github.com/go-chi/chi v4.0.2+incompatible
github.com/gofrs/uuid v4.4.0+incompatible
Copy link
Contributor Author

@Flashy78 Flashy78 Mar 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows for passing in a search query that can be a name or a stash id, and parse the text to find out which one.
It's the same library used on Stashbox.

github.com/golang-jwt/jwt/v4 v4.0.0
github.com/golang-migrate/migrate/v4 v4.15.0-beta.1
github.com/gorilla/securecookie v1.1.1
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,8 @@ github.com/gocql/gocql v0.0.0-20190301043612-f6df8288f9b4/go.mod h1:4Fw1eo5iaEhD
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gofrs/uuid v3.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gofrs/uuid v4.0.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gofrs/uuid v4.4.0+incompatible h1:3qXRTX8/NbyulANqlc0lchS1gqAVxRgsuW1YrTJupqA=
github.com/gofrs/uuid v4.4.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.0.0 h1:RAqyYixv1p7uEnocuy8P1nru5wprCh/MH2BIlW5z5/o=
Expand Down
4 changes: 2 additions & 2 deletions gqlgen.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,8 @@ models:
model: github.com/stashapp/stash/internal/manager.AutoTagMetadataInput
CleanMetadataInput:
model: github.com/stashapp/stash/internal/manager.CleanMetadataInput
StashBoxBatchPerformerTagInput:
model: github.com/stashapp/stash/internal/manager.StashBoxBatchPerformerTagInput
StashBoxBatchTagInput:
model: github.com/stashapp/stash/internal/manager.StashBoxBatchTagInput
SceneStreamEndpoint:
model: github.com/stashapp/stash/internal/manager.SceneStreamEndpoint
ExportObjectTypeInput:
Expand Down
23 changes: 23 additions & 0 deletions graphql/documents/data/scrapers.graphql
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
fragment ScrapedStudioData on ScrapedStudio {
stored_id
name
url
parent {
stored_id
name
url
image
remote_site_id
}
image
remote_site_id
}

fragment ScrapedPerformerData on ScrapedPerformer {
stored_id
name
Expand Down Expand Up @@ -97,6 +112,14 @@ fragment ScrapedSceneStudioData on ScrapedStudio {
stored_id
name
url
parent {
stored_id
name
url
image
remote_site_id
}
image
remote_site_id
}

Expand Down
6 changes: 5 additions & 1 deletion graphql/documents/mutations/stash-box.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,14 @@ mutation SubmitStashBoxFingerprints($input: StashBoxFingerprintSubmissionInput!)
submitStashBoxFingerprints(input: $input)
}

mutation StashBoxBatchPerformerTag($input: StashBoxBatchPerformerTagInput!) {
mutation StashBoxBatchPerformerTag($input: StashBoxBatchTagInput!) {
stashBoxBatchPerformerTag(input: $input)
}

mutation StashBoxBatchStudioTag($input: StashBoxBatchTagInput!) {
stashBoxBatchStudioTag(input: $input)
}

mutation SubmitStashBoxSceneDraft($input: StashBoxDraftSubmissionInput!) {
submitStashBoxSceneDraft(input: $input)
}
Expand Down
6 changes: 6 additions & 0 deletions graphql/documents/queries/scrapers/scrapers.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ query ListMovieScrapers {
}
}

query ScrapeSingleStudio($source: ScraperSourceInput!, $input: ScrapeSingleStudioInput!) {
scrapeSingleStudio(source: $source, input: $input) {
...ScrapedStudioData
}
}

query ScrapeSinglePerformer($source: ScraperSourceInput!, $input: ScrapeSinglePerformerInput!) {
scrapeSinglePerformer(source: $source, input: $input) {
...ScrapedPerformerData
Expand Down
7 changes: 6 additions & 1 deletion graphql/schema/schema.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,9 @@ type Query {
"""Scrape for multiple scenes"""
scrapeMultiScenes(source: ScraperSourceInput!, input: ScrapeMultiScenesInput!): [[ScrapedScene!]!]!

"""Scrape for a single studio"""
scrapeSingleStudio(source: ScraperSourceInput!, input: ScrapeSingleStudioInput!): [ScrapedStudio!]!

"""Scrape for a single performer"""
scrapeSinglePerformer(source: ScraperSourceInput!, input: ScrapeSinglePerformerInput!): [ScrapedPerformer!]!
"""Scrape for multiple performers"""
Expand Down Expand Up @@ -311,7 +314,9 @@ type Mutation {
backupDatabase(input: BackupDatabaseInput!): String

"""Run batch performer tag task. Returns the job ID."""
stashBoxBatchPerformerTag(input: StashBoxBatchPerformerTagInput!): String!
stashBoxBatchPerformerTag(input: StashBoxBatchTagInput!): String!
"""Run batch studio tag task. Returns the job ID."""
stashBoxBatchStudioTag(input: StashBoxBatchTagInput!): String!

"""Enables DLNA for an optional duration. Has no effect if DLNA is enabled by default"""
enableDLNA(input: EnableDLNAInput!): Boolean!
Expand Down
35 changes: 26 additions & 9 deletions graphql/schema/types/scraper.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,21 @@ type ScrapedStudio {
stored_id: ID
name: String!
url: String
parent: ScrapedStudio
image: String

remote_site_id: String
}

type ScrapedParentStudio {
stored_id: ID
name: String!
url: String
image: String

remote_site_id: String
}
Flashy78 marked this conversation as resolved.
Show resolved Hide resolved

type ScrapedTag {
"""Set if tag matched"""
stored_id: ID
Expand Down Expand Up @@ -147,6 +157,11 @@ input ScrapeMultiScenesInput {
scene_ids: [ID!]
}

input ScrapeSingleStudioInput {
"""Query can be either a name or a Stash ID"""
query: String
}

input ScrapeSinglePerformerInput {
"""Instructs to query by string"""
query: String
Expand Down Expand Up @@ -208,16 +223,18 @@ type StashBoxFingerprint {
duration: Int!
}

"""If neither performer_ids nor performer_names are set, tag all performers"""
input StashBoxBatchPerformerTagInput {
"Stash endpoint to use for the performer tagging"
"""If neither ids nor names are set, tag all items"""
input StashBoxBatchTagInput {
"Stash endpoint to use for the tagging"
endpoint: Int!
"Fields to exclude when executing the performer tagging"
"Fields to exclude when executing the tagging"
exclude_fields: [String!]
"Refresh performers already tagged by StashBox if true. Only tag performers with no StashBox tagging if false"
"Refresh items already tagged by StashBox if true. Only tag items with no StashBox tagging if false"
refresh: Boolean!
"If set, only tag these performer ids"
performer_ids: [ID!]
"If set, only tag these performer names"
performer_names: [String!]
Flashy78 marked this conversation as resolved.
Show resolved Hide resolved
"If batch adding studios, should their parent studios also be created?"
createParent: Boolean!
"If set, only tag these ids"
ids: [ID!]
"If set, only tag these names"
names: [String!]
}
2 changes: 2 additions & 0 deletions graphql/schema/types/studio.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ input StudioCreateInput {
name: String!
url: String
parent_id: ID
parent: StudioCreateInput
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the way this is processed in the resolver logic, I don't think it's a good design to include this. The behaviour isn't clear from the outside and the behaviour is based on whether an existing studio exists or not. The client should be responsible for creating/updating the parent studio. The only benefit I see in combining these operations is that you end up with a single transaction to roll back if needed, but I don't think it's worth it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved all parent studio logic out of the sqlite package.
I think I was trying to treat it logically as a single operation (not necessarily a single transaction). I wanted to avoid duplicating all of the parent matching/creating/updating logic in all of the multiple places where studios are created (identify/import/batch update/tagger view).
All of that code duplication can lead to bugs in the future when there's a new way to create a studio but all of the parent logic is not copied exactly. Like when disambiguation was added, it was added to all of the various places you can create a performer, but missed in identify.

I think this comment means you're suggesting to also write this parent logic in the frontend, so that the frontend can make an api call to create/update a parent studio, then attach it to the main studio and make another api call to create/update studio?

"""This should be a URL or a base64 encoded data URL"""
image: String
stash_ids: [StashIDInput!]
Expand All @@ -46,6 +47,7 @@ input StudioUpdateInput {
name: String
url: String
parent_id: ID,
parent: StudioCreateInput
"""This should be a URL or a base64 encoded data URL"""
image: String
stash_ids: [StashIDInput!]
Expand Down
10 changes: 10 additions & 0 deletions graphql/stash-box/query.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ fragment StudioFragment on Studio {
urls {
...URLFragment
}
parent {
name
id
}
images {
...ImageFragment
}
Expand Down Expand Up @@ -161,6 +165,12 @@ query FindSceneByID($id: ID!) {
}
}

query FindStudio($id: ID, $name: String) {
findStudio(id: $id, name: $name) {
...StudioFragment
}
}

mutation SubmitFingerprint($input: FingerprintSubmission!) {
submitFingerprint(input: $input)
}
Expand Down
82 changes: 27 additions & 55 deletions internal/api/resolver_model_studio.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,14 @@ package api

import (
"context"
"time"

"github.com/stashapp/stash/internal/api/loaders"
"github.com/stashapp/stash/internal/api/urlbuilders"
"github.com/stashapp/stash/pkg/gallery"
"github.com/stashapp/stash/pkg/image"
"github.com/stashapp/stash/pkg/models"
"github.com/stashapp/stash/pkg/performer"
)

func (r *studioResolver) Name(ctx context.Context, obj *models.Studio) (string, error) {
if obj.Name.Valid {
return obj.Name.String, nil
}
panic("null name") // TODO make name required
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name is required in the UI, but I don't think it's enforced anywhere. I will make a note to address that later.

}

func (r *studioResolver) URL(ctx context.Context, obj *models.Studio) (*string, error) {
if obj.URL.Valid {
return &obj.URL.String, nil
}
return nil, nil
}

func (r *studioResolver) ImagePath(ctx context.Context, obj *models.Studio) (*string, error) {
baseURL, _ := ctx.Value(BaseURLCtxKey).(string)
imagePath := urlbuilders.NewStudioURLBuilder(baseURL, obj).GetStudioImageURL()
Expand All @@ -47,15 +31,16 @@ func (r *studioResolver) ImagePath(ctx context.Context, obj *models.Studio) (*st
return &imagePath, nil
}

func (r *studioResolver) Aliases(ctx context.Context, obj *models.Studio) (ret []string, err error) {
if err := r.withReadTxn(ctx, func(ctx context.Context) error {
ret, err = r.repository.Studio.GetAliases(ctx, obj.ID)
return err
}); err != nil {
return nil, err
func (r *studioResolver) Aliases(ctx context.Context, obj *models.Studio) ([]string, error) {
if !obj.Aliases.Loaded() {
if err := r.withTxn(ctx, func(ctx context.Context) error {
return obj.LoadAliases(ctx, r.repository.Studio)
}); err != nil {
return nil, err
}
}

return ret, err
return obj.Aliases.List(), nil
}

func (r *studioResolver) SceneCount(ctx context.Context, obj *models.Studio) (ret *int, err error) {
Expand Down Expand Up @@ -107,16 +92,12 @@ func (r *studioResolver) PerformerCount(ctx context.Context, obj *models.Studio)
}

func (r *studioResolver) ParentStudio(ctx context.Context, obj *models.Studio) (ret *models.Studio, err error) {
if !obj.ParentID.Valid {
if obj.ParentID == nil {
return nil, nil
}

return loaders.From(ctx).StudioByID.Load(int(obj.ParentID.Int64))
}

func (r *studioResolver) ChildStudios(ctx context.Context, obj *models.Studio) (ret []*models.Studio, err error) {
if err := r.withReadTxn(ctx, func(ctx context.Context) error {
ret, err = r.repository.Studio.FindChildren(ctx, obj.ID)
ret, err = r.repository.Studio.Find(ctx, *obj.ParentID)
return err
}); err != nil {
return nil, err
Expand All @@ -125,48 +106,39 @@ func (r *studioResolver) ChildStudios(ctx context.Context, obj *models.Studio) (
return ret, nil
}

func (r *studioResolver) StashIds(ctx context.Context, obj *models.Studio) ([]*models.StashID, error) {
var ret []models.StashID
func (r *studioResolver) ChildStudios(ctx context.Context, obj *models.Studio) (ret []*models.Studio, err error) {
if err := r.withReadTxn(ctx, func(ctx context.Context) error {
var err error
ret, err = r.repository.Studio.GetStashIDs(ctx, obj.ID)
ret, err = r.repository.Studio.FindChildren(ctx, obj.ID)
return err
}); err != nil {
return nil, err
}

return stashIDsSliceToPtrSlice(ret), nil
return ret, nil
}

func (r *studioResolver) Rating(ctx context.Context, obj *models.Studio) (*int, error) {
if obj.Rating.Valid {
rating := models.Rating100To5(int(obj.Rating.Int64))
return &rating, nil
func (r *studioResolver) StashIds(ctx context.Context, obj *models.Studio) ([]*models.StashID, error) {
if !obj.StashIDs.Loaded() {
if err := r.withTxn(ctx, func(ctx context.Context) error {
return obj.LoadStashIDs(ctx, r.repository.Studio)
}); err != nil {
return nil, err
}
}
return nil, nil
}

func (r *studioResolver) Rating100(ctx context.Context, obj *models.Studio) (*int, error) {
if obj.Rating.Valid {
rating := int(obj.Rating.Int64)
return &rating, nil
}
return nil, nil
return stashIDsSliceToPtrSlice(obj.StashIDs.List()), nil
}

func (r *studioResolver) Details(ctx context.Context, obj *models.Studio) (*string, error) {
if obj.Details.Valid {
return &obj.Details.String, nil
func (r *studioResolver) Rating(ctx context.Context, obj *models.Studio) (*int, error) {
if obj.Rating != nil {
rating := models.Rating100To5(*obj.Rating)
return &rating, nil
}
return nil, nil
}

func (r *studioResolver) CreatedAt(ctx context.Context, obj *models.Studio) (*time.Time, error) {
return &obj.CreatedAt.Timestamp, nil
}

func (r *studioResolver) UpdatedAt(ctx context.Context, obj *models.Studio) (*time.Time, error) {
return &obj.UpdatedAt.Timestamp, nil
func (r *studioResolver) Rating100(ctx context.Context, obj *models.Studio) (*int, error) {
return obj.Rating, nil
}

func (r *studioResolver) Movies(ctx context.Context, obj *models.Studio) (ret []*models.Movie, err error) {
Expand Down
7 changes: 6 additions & 1 deletion internal/api/resolver_mutation_stash_box.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,16 @@ func (r *mutationResolver) SubmitStashBoxFingerprints(ctx context.Context, input
return client.SubmitStashBoxFingerprints(ctx, input.SceneIds, boxes[input.StashBoxIndex].Endpoint)
}

func (r *mutationResolver) StashBoxBatchPerformerTag(ctx context.Context, input manager.StashBoxBatchPerformerTagInput) (string, error) {
func (r *mutationResolver) StashBoxBatchPerformerTag(ctx context.Context, input manager.StashBoxBatchTagInput) (string, error) {
jobID := manager.GetInstance().StashBoxBatchPerformerTag(ctx, input)
return strconv.Itoa(jobID), nil
}

func (r *mutationResolver) StashBoxBatchStudioTag(ctx context.Context, input manager.StashBoxBatchTagInput) (string, error) {
jobID := manager.GetInstance().StashBoxBatchStudioTag(ctx, input)
return strconv.Itoa(jobID), nil
}

func (r *mutationResolver) SubmitStashBoxSceneDraft(ctx context.Context, input StashBoxDraftSubmissionInput) (*string, error) {
boxes := config.GetInstance().GetStashBoxes()

Expand Down
Loading