fix: Retry writing lease when error code 429 (too many requests). #2773
Conversation
This should fix the issue. We are retrying here as suggested by GCS since writing the same object again and again is rate limited to once per second. Another thing that might help is writing to a new "temp" object every time (instead of using the same one for the same process). Fixes: https://github.com/GlareDB/cloud/issues/2746 Signed-off-by: Vaibhav <vrongmeal@gmail.com>
| // TODO: Generate a new temp path everytime? This will definitely avoid | ||
| // the rate limitation over over-writing the same object here and might | ||
| // just be enough to delay the "renaming" process so we don't exceed | ||
| // the limit there as well. |
There was a problem hiding this comment.
@scsmithr how do you feel about this?
There was a problem hiding this comment.
I think that'd be reasonable, we can just generate a temp path every time. For some reason, I had read the original error message thinking that we were errorring on writing to the actual lease, and not the temp. But if it's just the temp, then this seems good.
There was a problem hiding this comment.
I merged this PR before reading this. Will open another to add this as well.
| // TODO: Generate a new temp path everytime? This will definitely avoid | ||
| // the rate limitation over over-writing the same object here and might | ||
| // just be enough to delay the "renaming" process so we don't exceed | ||
| // the limit there as well. |
There was a problem hiding this comment.
I think that'd be reasonable, we can just generate a temp path every time. For some reason, I had read the original error message thinking that we were errorring on writing to the actual lease, and not the temp. But if it's just the temp, then this seems good.
| // Error code 429 is not handled by object_store's retry | ||
| // configuration. We should maybe upstream to catch this | ||
| // and retry properly. | ||
| if store == &"GCS" && source.contains("429") { |
There was a problem hiding this comment.
I worry about the fragility of this little block, (in terms of the matching/destructuring of the error as well as the constants/string parsing of the error.
This should fix the issue. We are retrying here as suggested by GCS since writing the same object again and again is rate limited to once per second.
Another thing that might help is writing to a new "temp" object every time (instead of using the same one for the same process).
Fixes: https://github.com/GlareDB/cloud/issues/2746