Store image groupings in sidecar file (XMP)

[har0ke]

Is your feature request related to a problem? Please describe.

Grouping information is not stored in the sidecar files. When importing images into a new darktable database or removing and re-importing them, All manual applied groups are lost. Is there a reason this is currently not done?

Describe the solution you'd like
Instead of just storing the group parent in the database, also store it in the sidecar file.

Alternatives
Document all metadata that is not stored in the sidecar files. The documentation page about sidecar files would be my preferred location for such information: https://docs.darktable.org/usermanual/4.6/en/overview/sidecar-files/sidecar/ .

Additional context
Darktable removed all my film-rolls from the database (possibly because I forgot to mount my volume containing my images and sidecar files). I thought simple re-importing would recover all images and metadata. But sadly, not all groups were recovered. In my case, I had a recent backup of my library.db, but that might not always and for everybody be the case.

[ralfbrown]

Likely to be difficult to implement - grouping is stored in the database by image ID, which is a simple number. Re-importing after removal (or importing into a new database) gives the image a new ID, since they are just consecutive numbers, incrementing for each image which is imported.

BTW, if you get skulls displayed because you forgot to mount a drive with images, all you need to do is mount the drive and restart darktable. Unless you specifically tell it to delete images (either by explicit selection or using one of the "purge missing" scripts), dt doesn't remove things from the database. They'll be there as soon as the files themselves are accessible again.

[har0ke]

To just save relative file names is probably not enough, because you can also group across film-rolls, correct? I think grouping within film-rolls is the most common use case. But only handling that use case could also result in confusion...

As far as you know, is this the only metadata, that is not saved in the sidecar file?

[har0ke]

BTW, if you get skulls displayed because you forgot to mount a drive with images, all you need to do is mount the drive and restart darktable. Unless you specifically tell it to delete images (either by explicit selection or using one of the "purge missing" scripts), dt doesn't remove things from the database. They'll be there as soon as the files themselves are accessible again.

Yes, usually this is what happens and I have never had this issue before. I am not entirely sure why darktable removed my files from the database and I cannot reproduce it.

[ralfbrown]

I'm not terribly familiar with that part of the codebase, but as grouping is pretty much the only metadata that involves multiple images (things like filmrolls are still per-image even though many may share the same value), there's a good chance that that is in fact the only metadata not stored in sidecars.

[rekcodocker]

A way to implement this is to generate a unique ID whenever a group is created.
Store that ID in the XMP files of all the images that belong to that group.
When you add a new image to an existing group you already have the group-ID, it just needs to be added to the DB and the XMP file.

Notes:

• the XMP would have a section like: and then a list of group-IDs.
• Some images can belong to multiple groups, that should not be a problem.
• to generate a unique ID possibly use date/time and a unique number.
• When you import a new folder, all images which are part of an existing group are recognized as such and are added to the group. That means that restoring your database requires you to import the folders. The group might be incomplete until you have imported all the folders and you would not know it was incomplete. In the usecase of the Isssue-issuer, this would not be a problem: he wants the groups restored on import.
• if could be possible that a future version of DT gets a function to clean up 'stale' groups which have only one member.

[wpferguson]

Some images can belong to multiple groups, that should not be a problem.

Until you try and get the group leader. If the image is stored in multiple groups which group leader would be correct? This would break things. You could use tags to implement multiple groups and that would exist across re-imports.

Store that ID in the XMP files of all the images that belong to that group.

If someone sends you an XMP file and you apply it, it's grouping metadata could corrupt your grouping data.

clean up 'stale' groups which have only one member.

All new images when imported are a group of one, i.e. their id matches their group_id. So the definition of a group (at least database wise) is a group_id with a count > 1.

A way to implement this is to generate a unique ID whenever a group is created.

Wouldn't the ID only be unique to the environment it was generated in? If I imported those XMP files into another existing environment could there be a possibility of an collision?

As I was typing the comments up above, I thought of implementing multiple groups with tags. But, that would also be a way to persist a group since tags are added to the XMP files. We could use something like darktable|group|xxxxx and darktable|group-leader|xxxxx since those don't show up in the normal tags display. After import a job could run to populate the group_id fields in the database based on the tags.

[wpferguson]

Using tags would also be a way to persist any other metadata that isn't in an XMP file, i.e. darktable|metadata|<name>|<content>.

[rekcodocker]

Until you try and get the group leader. If the image is stored in multiple groups which group leader would be correct? This would break things. You could use tags to implement multiple groups and that would exist across re-imports.

Attach a preference-number to the group-ID. The image in the group with the highest number becomes the group leader. This can make interesting behaviour if you only import part of a group.
This is only a thing in edge cases. You need a few rules to always end up with a single group leader, but the DT always allows the user to change it.

If someone sends you an XMP file and you apply it, it's grouping metadata could corrupt your grouping data.

Good point. The ID needs to be really unique. You would simply receive the information that this image was apparently important enough to belong to a group, and you would also get that group.

Wouldn't the ID only be unique to the environment it was generated in? If I imported those XMP files into another existing environment could there be a possibility of an collision?

Yes. You'd need a UUID (universal unique id). This concept has been worked out, no need to re-invent it.

As I was typing the comments up above, I thought of implementing multiple groups with tags. But, that would also be a way to persist a group since tags are added to the XMP files. We could use something like darktable|group|xxxxx and darktable|group-leader|xxxxx since those don't show up in the normal tags display. After import a job could run to populate the group_id fields in the database based on the tags.

Hmmm interesting.
The tags also end up in the XMP file. So you agree with the idea, only you would store it in the 'tag' section and not in a new 'group' section.

[wpferguson]

So you agree with the idea

I neither agree or disagree. I was simply trying to figure out a solution that:

• wouldn't break grouping (and it's surrounding infrastructure) as it is now
• could reasonably be implemented without taking years
• could be built on what already exists (i.e. tags, signals), so less code change
• might provide a solution for other data that are needed|wanted to be persisted across multiple darktable installations

The image in the group with the highest number becomes the group leader

That's not always the desired behavior. In RAW+JPG|HIF groupings some users want the second image to be the group leader.

if you only import part of a group

If you implement group persistence with tags, you could just wait until the group leader gets imported, then query the tags for the group members and rebuild the complete group.

[wpferguson]

Have a working solution that:

• didn't break grouping (and it's surrounding infrastructure) as it is now
• only took a week
• built on what already exists. Added a grouping info changed event, triggered from the grouping code, a UUID function to the Lua API and a couple hundred lines of Lua code. Tags are added/removed/changed as groups are added/removed/changed. The tags are saved to the XMP files. On import to a new/different instance the groups are rebuilt. It's all event driven, so you can turn it on and forget it.