-
Notifications
You must be signed in to change notification settings - Fork 181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upload failures due to locking issue #5024
Comments
@rhafer Can you update this ticket with your findings? |
with #5061 we no longer calculate the dir size, which also reduces the amount of readlocks we have to create. |
I can confirm that the behavior with latest master is a lot better. But I am still able getting the locking error. When uploading 1000 files (in a single directory) via the web frontend. (on an ocis server backed by nfs storage):
This happens a lot less often now. For rc1 I get about 50 failed uploads, for latest master it's 0-3 failed uploads. |
I've been experimenting a bit with the locking timeout and retry values in https://github.com/cs3org/reva/blob/edge/pkg/storage/utils/filelocks/filelocks.go as the default of 3ms seemed a bit low. Increasing that timeout helped a lot. I tried with 30ms (not sure if that value makes any good sense) and didn't see lock failures anymore. It might make sense to add configuration knobs for the filelocking timeout and retries. (I assume the behavior will depend a lot on the underlying storage) |
we made the lock cycle duration factor configurable and increased it in cs3org/reva#3493 |
Reva update done. |
Describe the bug
In a test environment where I placed the filesystem for decomposedfs on a remote NFS volume I see quite a few locking issue when uploading a small folder hierarchy (3 levels deep, about 500MB, 550 files in 300 directories) via the WebUI. This seems to be caused by lock contention (like we already saw before):
This was when testing with v2.0.0-rc.1 the behavior is much better when compliing against latest reva/edge (because cs3org/reva#3397 improved the locking behavior). So it might actually be a non-issue 🤞 once we bumped reva for the next RC.
An easy way to reproduce this, seem to be to create a deep folder structure, with a couple of files at the deepest level and upload that folder via the webui (use some NFS share for ocis strorage-users):
The text was updated successfully, but these errors were encountered: