-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BigQuery, Storage]: resumable uploads fail with non-retryable GoogleAPICallError: 410 PUT Service Temporarily Unavailable #7530
Comments
Wow, that is definitely the wrong error response:
@tswast Can you check with the back-end team for the source of this error response? |
Yikes. Filed bug 128935544 internally. |
This is a service issue. I don't think we should keep this issue open. |
There is discussion on internal bug 115694647 as well as public issue https://issuetracker.google.com/137168102 This issue affects all services. There are cases where one of the API calls for a resumable upload get stuck and the upload cannot be continued. The client would need to start the upload from the beginning when such a failure happens, but currently does not due to retries being handled at the request level. Some manual logic would be needed to solve this. We might even need to generate a new job ID, though. Not sure how far BigQuery gets into job creation if the upload fails. |
@tswast I'm not sure how we are supposed to deal with the The "gsutil Retry Handling Strategy" doc does not include 410 as one of the transient failures for which automatic retry is appropriate. |
410 is correct, actually. There is an internal "upload" resource that has had an irrecoverable failure. The only way to recover is by starting the upload flow with a fresh request from the very beginning of the file / stream. |
Supposedly, BigQuery doesn't create the job, so we should be able to use the same job ID. |
Not according to RFC 2616, which states:
The target for the PUT request is That the server crapped out somehow while trying to process the One could surely argue that the API should require a |
The point I'm trying to make is that we can't retry the request identically. Resumable uploads are a multi-request operation. This failure occurs partway through an upload. To retry the upload, we need to seek back to the beginning of the file and start the whole process over. |
Closing in favor of a fresh feature request. Thanks everyone for the discussion. |
OS: Linux dc32b7e8763a 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux
Python version: Python 2.7.6
google-cloud-bigquery: 1.8.0
We're getting a new flake we've never seen before. Furthermore, the load_table_from_file function doesn't take any retry parameters, so we can't patch this on our end unless we want to implement our own retry.
Code sample:
Stack trace:
The text was updated successfully, but these errors were encountered: