Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

don't download photos of shared albums #432

Closed
rui-nar opened this issue May 10, 2023 · 18 comments
Closed

don't download photos of shared albums #432

rui-nar opened this issue May 10, 2023 · 18 comments

Comments

@rui-nar
Copy link
Contributor

rui-nar commented May 10, 2023

Hi
Is it possible not to download the photos of shared albums when using the --skip-shared-albums option?

My usecase is the following: I have an automatic backup of the photos taken with my smartphone in original quality with an app into my NAS. Since I do my sorting & deleting from Google Photos, I'm looking to "synchronise" my local copy with the "truth" which is my Google Photos referential (I'm looking into using the --compare-folder option) but first I need a proper mirror of Google Photos into my NAS of ONLY my photos and my albums .. not shared albums or other people's photos ...

I hope I was clear
Rui
@gilesknap
Copy link
Owner

As far as I recall, --skip-shared-albums works as you require. I don't use it on my own library but I do have a system test that verifies that it works (at least on the test account).

Using --compare-folder successfully may be pretty hard. It is probably over complicated by my attempts to cope with cameras from the 90s that frequently used the same filenames and often trashed any metadata. That's why I deprecated it.

Best of luck!

@rui-nar
Copy link
Contributor Author

rui-nar commented May 10, 2023 via email

@gilesknap
Copy link
Owner

Sure I'll verify against my account (which has many shared albums). I'll get back to you this weekend hopefully.

@rui-nar
Copy link
Contributor Author

rui-nar commented May 15, 2023

still interested on the results of your tests btw ;)

@gilesknap
Copy link
Owner

Sorry. Will be able to look at this today.

@gilesknap
Copy link
Owner

gilesknap commented May 16, 2023 •
edited
Loading

Logging of tests

using 2023 photos only.
According to Google Photos Web I have:

  • 1940 photos
  • 7 albums (one of which is auto)
  • 2 named shared albums
  • 1 shared album contains 145 photos not mine
  • CORRECTION: there is a 2nd shared album with 45 photos not mine
  • 1 shared album contains 15 mine, 1 not mine CORRECTION: this is a unamed album and does not count
  • there are other unamed shared folders (which should be ignored - TODO verify this)

Test 1 - with no shared

(venv) [giles@ws1 gphoto-test]$ gphotos-sync no-shared/ --start-date 2023-01-01 --skip-shared-albums --progress
05-16 14:07:29 WARNING  gphotos-sync 3.1.3.dev0+ga631449.d20230516 2023-05-16 14:07:29.504828 
 5-16 14:07:29 WARNING  Indexing Google Photos Files ... 
05-16 14:07:58 WARNING  indexed 1577 items ..
 5-16 14:07:58 WARNING  Downloading Photos ... 
05-16 14:15:46 WARNING  Downloaded 1940 Items, Failed 0, Already Downloaded 0 
 5-16 14:15:46 WARNING  Indexing Albums ... 
05-16 14:35:04 WARNING  Indexed 468 Albums ..
05-16 14:35:04 WARNING  Downloading Photos ... 
05-16 14:35:05 WARNING  Downloaded 1943 Items, Failed 0, Already Downloaded 1940 
05-16 14:35:05 WARNING  Creating album folder links to media ... 
05-16 14:35:23 WARNING  Created 893 new album folder links 
05-16 14:35:23 WARNING  Done.

Initial number of images is 1940 as expected
TODO: 3 extra files came down with albums - investigate

Albums created:

(venv) [giles@ws1 no-shared]$ ls albums/* | wc
    515    1991   12248
(venv) [giles@ws1 no-shared]$ ls albums/1999/0803\ Trip\ to\ San\ Francisco\ and\ Las\ Vegas/
(venv) [giles@ws1 no-shared]$ 
          
(venv) [giles@ws1 no-shared]$ ls albums/2023/0215\ Latest\ works\ by\ Noah/
'0141_Scanned Document-3.jpg'   0150_swim.jpg                                                 '0159_Scanned Document-1.jpg'
'0142_Scanned Document-7.jpg'   0151_awards-2.jpg                                             '0160_Scanned Document-2.jpg'
'0143_Scanned Document-5.jpg'   0152_awards-1.jpg                                              0161_IMG_20230202_0001.jpg
'0144_Scanned Document-9.jpg'   0153_1673704825830-42118ba4-ef47-4bd9-9f5c-be186620fc29_.jpg   0162_PXL_20230207_222258401.jpg
'0145_Scanned Document-8.jpg'   0154_1673704893896-aa4f079a-eb5c-4ba9-8f99-62ba9357e409.jpg    0163_new-2.jpg
'0146_Scanned Document-4.jpg'   0155_1673802139435-ed51621e-6a77-4797-a49c-78c01459e36d_.jpg   0164_new-1.jpg
'0147_Scanned Document-1.jpg'   0156_IMG_6789.HEIC                                             0165_new2.jpg
'0148_Scanned Document-6.jpg'   0157_PXL_20230127_170708525.jpg                                0166_PXL_20230215_223643463.jpg
'0149_Scanned Document-2.jpg'   0158_Beshianlong.jpg                                           0167_PXL_20230215_223807405.jpg
(venv) [giles@ws1 no-shared]$

Hmm - a links folder was created for every album in my library - but only files dated past 2023 are linked (so that is a lot of empty folders)

Albums in 2023 is 7 as expected

(venv) [giles@ws1 no-shared]$ ls albums/2023/0
0115 Birdworld - meeting Emus/             0326 Hockney/                              0515 Millie!/
0215 Latest works by Noah/                 0416 Dinosnores at Natural History Museum/ 
0218 Jurassic Coast (West Lulworth)/       0418 Ivy's 11th Birthday/

@rui-nar
Copy link
Contributor Author

rui-nar commented May 16, 2023 via email

@gilesknap
Copy link
Owner

Test 2 with Shared Albums

(venv) [giles@ws1 gphoto-test]$ gphotos-sync with-shared/ --start-date 2023-01-01  --progress
05-16 14:25:57 WARNING  gphotos-sync 3.1.3.dev0+ga631449.d20230516 2023-05-16 14:25:57.530072 
 5-16 14:25:57 WARNING  Indexing Google Photos Files ... 
05-16 14:26:31 WARNING  indexed 1940 items ..
 5-16 14:26:31 WARNING  Downloading Photos ... 
05-16 14:31:57 WARNING  Downloaded 1940 Items, Failed 0, Already Downloaded 0 
 5-16 14:31:57 WARNING  Indexing Shared (titled) Albums ... 
05-16 14:35:28 WARNING  Indexed 485 Shared (titled) Albums ..
 5-16 14:35:28 WARNING  Indexing Albums ... 
05-16 14:51:19 WARNING  Indexed 468 Albums ..
 5-16 14:51:19 WARNING  Downloading Photos ... 
05-16 14:52:13 WARNING  Downloaded 2130 Items, Failed 0, Already Downloaded 1940 
05-16 14:52:13 WARNING  Creating album folder links to media ... 
05-16 14:52:32 WARNING  Created 1115 new album folder links 
05-16 14:52:32 WARNING  Done.

TODO: back later to analyse these...

@gilesknap
Copy link
Owner

@rui-nar I don't have any immediate answers. But I now have two folders of files to do a detailed investigation with. I have to run now but will get back to this later today.

@gilesknap
Copy link
Owner

OK - I do have an answer for the 1577 vs 1940. This was just because I ran and aborted gphotos-sync in that folder before the test and it had already started indexing when I aborted (I wanted to restart with --progress). You can see the correct numbers in Test 2.

back in a couple of hours.

@gilesknap
Copy link
Owner

with-shared downloaded an extra 2130-1940 = 190 files whereas I thought I had 146 shared files.

listing the files in the extra album folders that appear in albums/2023 gives 227 extra files.

Neither of these figures add up.

@rui-nar
Copy link
Contributor Author

rui-nar commented May 16, 2023 via email

@gilesknap
Copy link
Owner

gilesknap commented May 16, 2023 •
edited
Loading

Albums that appear in the shared sync:

(venv) [giles@ws1 with-shared]$ ls -l albums/2023
total 88
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0115 Birdworld - meeting Emus'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0215 Latest works by Noah'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0218 Charmouth, Corfe Castle & Lyme Regis'
drwxr-xr-x 2 giles dialout 20480 May 16 15:28 '0218 Jurassic Coast (West Lulworth)'
drwxr-xr-x 2 giles dialout 12288 May 16 15:28 '0219 James 11th Birthday @ GoApe'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0326 Hockney'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0326 Hockney in Kings X'
drwxr-xr-x 2 giles dialout 12288 May 16 15:28 '0416 Dinosnores at Natural History Museum'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 "0418 Ivy's 11th Birthday"
drwxr-xr-x 2 giles dialout 20480 May 16 15:28 '0515 Millie!'

Albums that appear in non-shared sync

drwxr-xr-x 2 giles dialout  4096 May 16 14:35 '0115 Birdworld - meeting Emus'
drwxr-xr-x 2 giles dialout  4096 May 16 14:35 '0215 Latest works by Noah'
drwxr-xr-x 2 giles dialout 16384 May 16 14:35 '0218 Jurassic Coast (West Lulworth)'
drwxr-xr-x 2 giles dialout  4096 May 16 14:35 '0326 Hockney'
drwxr-xr-x 2 giles dialout 12288 May 16 14:35 '0416 Dinosnores at Natural History Museum'
drwxr-xr-x 2 giles dialout  4096 May 16 14:35 "0418 Ivy's 11th Birthday"
drwxr-xr-x 2 giles dialout 20480 May 16 14:35 '0515 Millie!'

EXTRA ALBUMS

drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0218 Charmouth, Corfe Castle & Lyme Regis'
drwxr-xr-x 2 giles dialout 12288 May 16 15:28 '0219 James 11th Birthday @ GoApe'
drwxr-xr-x 2 giles dialout  4096 May 16 15:28 '0326 Hockney in Kings X'

The first above is an auto generated album (whch apparently counts as a shared one even though its only in my library) so it contains links to pre-existing photos and can be ignored.

Second is fully shared with 145 images
Third is fully shared with 42 images

This adds up to 187 images extra which is 3 less than reported in the log (so the same 3 that the non-shared sync did probably).

Now checking the difference between actual photos downloaded in each case:

(venv) [giles@ws1 no-shared]$ find photos/ -type f | wc
   1943    1960   69317
(venv) [giles@ws1 no-shared]$ cd ../with-shared/
(venv) [giles@ws1 with-shared]$ find photos/ -type f | wc
   2130    2189   74915
(venv) [giles@ws1 with-shared]$

2130 - 1943 = 187 which is exactly right.

EDIT: forgot to say that this tallies with the photo count in these 2 albums in the Web GUI

@gilesknap
Copy link
Owner

Final Synopsis

For my library the 2023 shared/no-shared albums do behave as expected with 187 less photos when using --skip-shared-albums.

Things to note:

  • The code reports downloading an extra 3 files when scanning (non-shared) albums and I do see those extra 3 in the local file count. This is probably one of the google API bugs where not all photos are delivered initially with a global scan see Google API Bugs #119. This could do with further investigation (but I doubt Google will respond to a new ticket on this)
  • There is confusion over what entails a shared album. Google's own creations show up as shared albums but groups of photos shared without creating a shared album do not.
  • gphotos-sync does scan all albums (of correct category shared/non-shared) and create empty folders for those whose contents don't match the date filter - it might be nice to fix that.

@gilesknap
Copy link
Owner

@rui-nar These tests were run against the main branch.

Let me know your results.

@rui-nar
Copy link
Contributor Author

rui-nar commented May 17, 2023

hey @gilesknap
I have to apologise Giles, I ran all the tests you did and I have the SAME results then you. So I re-ran all my previous commands (which I store in a text file) and all results were in line with expectations. I cannot explain this !!!
I know I upgraded the version to the latest at some point in time but I was sure that I had re-tested everything.

So sorry for having wasted your time.

Btw, the --compare-folder worked flawlessly even if I took some time to understand that the differences I was seeing were archived pictures in GPhotos which I can't seem to filter out in the Google Photos search.

Anyways. Awesome project !! Almost able to fulfill my wife's usecase ;)

@gilesknap
Copy link
Owner

Well it is possible that you simply got inconsistent behaviour from the Google API. I was reasonably convinced that this might have been broken so its good to have verified it properly.

Regarding the archived photos - there is a gphotos-sync option that is supposed to include them --archived. Otherwise they are not supposed to be included. Is that not your experience (this is probably for a different issue if you want to open one)

@rui-nar
Copy link
Contributor Author

rui-nar commented May 17, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gilesknap @rui-nar