Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests #320

Closed
qrkourier opened this issue Jul 29, 2021 · 5 comments
Closed

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests #320

qrkourier opened this issue Jul 29, 2021 · 5 comments
Assignees
@gilesknap
Labels
bug help wanted

Comments

@qrkourier
Copy link

Describe the bug
I'm unable to re-sync to download my newer photos despite having successfully done some in prior years. The search API is responding with a backoff error, and the 429-response branch isn't quite addressing this particular issue. I plan to work on the problem in my fork and to track progress and gather feedback here.

To Reproduce

~/Sites/gphotos-sync 429-reponse
# check out branch "429-response"
❯ pip install --editable .                                                               
Looking in indexes: https://pypi.org/simple, https://bitbucket-tools:****@netfoundry.jfrog.io/netfoundry/api/pypi/python/simple
Obtaining file:///home/kbingham/Sites/gphotos-sync
Requirement already satisfied: attrs in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (21.2.0)
Requirement already satisfied: exif in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (1.3.1)
Requirement already satisfied: appdirs in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (1.4.4)
Requirement already satisfied: requests_oauthlib in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (1.3.0)
Requirement already satisfied: pyyaml>=4.2b1 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (5.4.1)
Requirement already satisfied: psutil in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (5.8.0)
Requirement already satisfied: plum-py>=0.5.0 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from exif->gphotos-sync==2.14.0+5.g497c1ce) (0.7.0)
Requirement already satisfied: requests>=2.0.0 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (2.25.1)
Requirement already satisfied: oauthlib>=3.0.0 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (3.1.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (2021.5.30)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (1.26.5)
Requirement already satisfied: chardet<5,>=3.0.2 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (2.10)
Installing collected packages: gphotos-sync
  Attempting uninstall: gphotos-sync
    Found existing installation: gphotos-sync 2.14.0+5.g497c1ce
    Uninstalling gphotos-sync-2.14.0+5.g497c1ce:
      Successfully uninstalled gphotos-sync-2.14.0+5.g497c1ce
  Running setup.py develop for gphotos-sync
Successfully installed gphotos-sync-2.14.0+5.g497c1ce

~/Sites/gphotos-sync 429-reponse 
❯ pip show gphotos-sync
Name: gphotos-sync
Version: 2.14.0+5.g497c1ce
Summary: Google Photos and Albums backup tool
Home-page: https://github.com/gilesknap/gphotos-sync
Author: Giles Knap
Author-email: gilesknap@gmail.com
License: MIT
Location: /home/kbingham/Sites/gphotos-sync
Requires: attrs, exif, appdirs, requests-oauthlib, pyyaml, psutil
Required-by: 

❯ gphotos-sync --retry-download --max-threads 3 --log-level warn ~/Downloads/from_gphotos
07-28 22:52:03 WARNING  gphotos-sync 2.14.0+5.g497c1ce 2021-07-28 22:52:03.683065 
07-28 22:52:04 WARNING  Indexing Google Photos Files ... 
07-28 22:52:05 WARNING  indexed 6 items 
07-28 22:52:05 WARNING  Downloading Photos ... 
07-28 22:54:39 ERROR    FAILURE 1 downloading photos/2020/06/VID_20200604_201343.mp4 - 400 Client Error: Bad Request for url: https://video-downloads.googleusercontent.com/AGQNM9LFZ2qrswOXVFUH0ZnxMb51VQiQyZ-CDPk-4Z1eHp9QrT9_rD1pRenCLKpKCx_ulQ9HBGAwQ2zXfC_lccD1H96d2s9gUOk-k9F8sfJ0dY63fIGBmgG39W7VfVmJ3n8IePdbNXwimkLNQdPCk6f3eYozj8JTOTFrVoooy2XuiHyQw9hRM0esFapuQuSQDLNFhR9q-VKzda4RuezpB2YnzyCIaJJpb7_wfK05qJlIidLsAWg9ZoDYZHo15MM7KTA3540sLtfMZOynf7P5gO6tiWJg_Ml4f9BCTWbimAL5FV0wkVtlJyL6ijrpbFHW6g_9KVltrSk7X4FBaIjfYji9U1DuVUBeGvDz9DzqpcXqZdmF3fBfKQf0e9hgIBNAkbDF7ufMFG_23r46hxs5sjx9MAuQxA_UQXjkWmNFV6Bpta9YUC4UhxcGwHYRRb1_1WKH6YWxfAwiUUAc72u_ScJaF58X0GIzGzQpbIDKd_DvpImXY4fgZNwMYpEHJmBQ1mnNFa7u7V-v6UTKsW2BhOkcx1d8gH1L1EoVOfW6eYiykzndb4QhlHkRg5UtOAKx5jXDcl4zYcx2buHc5xb8QysTo09JA_bDFcSH-gcCKkwc1UBFZ5UjxOlKKdropGN2MLOiSEhhkIw_XG906riyRJgdw0x7d82YDkrxH0dd2SHMAFsNfPESFOTN0OtG1zlBCqi0w1RA2q6pM9XcgmUdQ7UckcHY0oe3lTvW9_wKlE1kfEJH6Gt4FcWoKbhoNjWjTUANuUhdQVrj5siqGn9nAHV2xQ4IYHYOi3x_4J28ngNWDvjQQ5RzUDXe-MR83phzPkpKJvxf82ItZtBzYQTo9sTQ80Bmlh55Hw_4fxoQ5wWmCp0UHtgHN-m19Ov-cbd1ZOjjnBeBI9xT_RrL0S7M2PCt0g0wHbf-mOCSrUzmV1BHGheRyPLQtXOxMlXQxGcGiZUjCEw7ANwy 
07-28 22:54:39 ERROR    FAILURE 2 downloading photos/2017/07/zelda-spooked-by-skates.mp4 - 400 Client Error: Bad Request for url: https://video-downloads.googleusercontent.com/AGQNM9JRfQxZa83qGToCMa3gn5CXNyrf4TveOxkNGVyCB7Gmq09kbPXjKB7X7NnxDLyGnZvBfjfK49TifCsPsVJapQZNIOjXm8tJWorpjxELs3Du09iNOqBKo_sDkble8PeAMwRAprAluGN6_uBiyE70zeRnZ58GCmaZe7NiareBLZXxFqHICeFKT9J1ZMc2HdEyyDcSmT8dak1ufCQXN3OYLUPnRHSs6qrFOjdHxMwfcw_BnRFV18Q85p0VE-7QAcTDdyXmFOMNTOb2jX0ATkGmYKwU49RjBKqzjQClWmK_rJlLV5TRLTTd0E_qJ1529rf0BZ15TEM1Q2d5dX4Yg-xMjTOFRDH2uf0_ffi8PC1o0HjBqjMh_yoqFdVuD95kwwomsGRHowTLN4A0JFwFLw2zyjQeanCrnEEg7_s1THwHyJ9y7NsFk-1GKUOcBNTfkVLC9zApYrEcAUnoPTv1qfUNeWwtgPTbCauk3bo27QeduALZDxprTB_4JFv4_RmKtxFI6Q12HNxDEk1LECkbsgK1pQFbjPPcXFzc13e67VPmktEMxnT-wVlREjLjSfSNlU_n22IRSZGfphUTVb1DDNQwMWYX3v_9FS90Gb0Rb91dQm52AatzWRBIKuaZGR1Dp3p7X_pIQDTypjlFrIp5nD5bHdJ8UAlzQWhro69_oSul2bsBl4gFZCct3YiYKmJmXpqWIN7UMDVCYJX8caP97P5KTRyaHvGz54O5oG4Y2ggZ-sicXphuv4EAgEZrtFhXAZRgWjTM0mPoiald2_Zyuayhe2lTRdhrraiY5PyoPq3-XkeFg81E_7qXpin1ot-CYJvrJ-3LboS7WYHcXSk3P6h9i9yr_2n4vc2vyKGlmZXT67PYB1emt9HTrG0Jjmo-7F2KITiaU5fzNGT_bNBaj8VAi0NbOPnZd4HdPipLmSXzz7bO3WVyQtnIeRHvJVNybjQO8lfuAYNJ 
07-28 22:54:39 WARNING  Downloaded 20 Items, Failed 2, Already Downloaded 26973 
07-28 22:54:39 WARNING  Indexing Shared (titled) Albums ... 
07-28 22:56:26 WARNING  Indexed 255 Shared (titled) Albums 
07-28 22:56:26 WARNING  Indexing Albums ... 
07-28 23:56:14 ERROR    
Process failed. 
Traceback (most recent call last):
  File "/home/kbingham/Sites/gphotos-sync/gphotos/Main.py", line 418, in main
    self.start(args)
  File "/home/kbingham/Sites/gphotos-sync/gphotos/Main.py", line 369, in start
    self.do_sync(args)
  File "/home/kbingham/Sites/gphotos-sync/gphotos/Main.py", line 344, in do_sync
    self.google_albums_sync.index_album_media()
  File "/home/kbingham/Sites/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 150, in index_album_media
    self.index_albums_type(
  File "/home/kbingham/Sites/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 203, in index_albums_type
    first_date, last_date = self.fetch_album_contents(
  File "/home/kbingham/Sites/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 75, in fetch_album_contents
    response = self._api.mediaItems.search.execute(body)
  File "/home/kbingham/Sites/gphotos-sync/gphotos/restclient.py", line 100, in execute
    result.raise_for_status()
  File "/home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://photoslibrary.googleapis.com/v1/mediaItems:search
07-28 23:56:14 WARNING  Done. 

# command ran for 1h 4m 11s
@gilesknap
Copy link
Owner

@qrkourier thanks this is very helpful.

I have been unable to test the backoff code in any meaningful way with my connection speed.

@qrkourier qrkourier mentioned this issue Jul 30, 2021
Closed
@synman
Copy link

synman commented Jul 30, 2021

when I got throttled on my big account, I let it sit idle for a full day then started back up with a max threads of 5. Delete your bad ids file and try again in a day or two with lower thread settings. worked for me.

@gilesknap
Copy link
Owner

@synman @qrkourier
UPDATE: I'm now using github actions for CI and it runs multiple targets simultaneously. This is causing me to see HTTP 429.

So I now have a way of testing this!

Once I get the next release out, I'll spend some time on this issue.

In the meantime one thing that helps is to use the option --threads 1 since it slows down the number of requests.

@gilesknap
Copy link
Owner

Looks like I fixed it. I'd forgotten to put 429 in the list of errors to do backoff for. Simple fix.

https://github.com/gilesknap/gphotos-sync/runs/6476970287?check_suite_focus=true

@synman @qrkourier Please can you try the HEAD of main to see if it works for your usescase?

Thanks

@gilesknap
Copy link
Owner

OK closing as fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gilesknap gilesknap

Labels
bug help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gilesknap @synman @qrkourier