-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Impossible to disk cleanup the registry (2.5.0) due to broken garbage cleanup? #1939
Comments
Can't really confirm anything without more information. It is possible that there was a still a tag or manifest which was referencing the layer, so seeing "Layer already exist" itself would not be indicative of an error. Are you having any other symptoms? You mentioned that that disk space was freed, but perhaps not as much as anticipated and it is filling up again too quickly? In addition to deleting the tags, are you also deleting manifests which may be untagged? |
Steps to reproduce:
Will try to give you my exact steps with curl next monday, but I think in the 988 ticket it is well explained. |
As seen in docker-archive/docker-registry#988 restarting the registry is needed and solves the problem. It might be complicated when running the registry in a docker container though, as stopping the server would stop the container and not allow to exec the GC in it... |
@eesprit by restarting the server in read-only mode, you mean creating a second config2.yml with this section as difference:
And then restarting the server with that readonly option, run GC, restart the server with the old config? Or is there an API way to tell the running server to put itself in readonly mode? |
I don't think there is an API call for that (the Docker Private Registry expose one, but this does not seem to exist in the distribution registry). So yes, I think the only way is to switch conf files. By the way, as far as I understand, the maintenance/readonly is under storage, not at the "root" of the config file. |
I am also able to reproduce this (exactly as zoobab says it). registry github.com/docker/distribution v2.5.1 # curl -kIXDELETE https://[registry]:5000/v2/docker.io/library/busybox/manifests/sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101
HTTP/1.1 202 Accepted
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Thu, 13 Oct 2016 11:39:31 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8
# docker run --rm registry:2 garbage-collect /etc/docker/registry/config.yml
time="2016-10-13T11:40:51Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/56/56bec22e355981d8ba0878c6c2f23b21f422f30ab0aba188b54f1ffeff59c190" go.version=go1.6.3 instance.id=f0bc6b6d-f7fa-4bb5-a63f-32f2e0ed8df0
time="2016-10-13T11:40:51Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/93/9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101" go.version=go1.6.3 instance.id=f0bc6b6d-f7fa-4bb5-a63f-32f2e0ed8df0
time="2016-10-13T11:40:51Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/e0/e02e811dd08fd49e7f6032625495118e63f597eb150403d02e3238af1df240ba" go.version=go1.6.3 instance.id=f0bc6b6d-f7fa-4bb5-a63f-32f2e0ed8df0
# curl -kXGET https://[registry]:5000/v2/docker.io/library/busybox/tags/list
{"name":"docker.io/library/busybox","tags":null}
# docker push [registry]:5000/docker.io/library/busybox
The push refers to a repository [[registry]:5000/docker.io/library/busybox]
e88b3f82283b: Layer already exists
latest: digest: sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101 size: 505
# curl -kXGET https://[registry]:5000/v2/docker.io/library/busybox/tags/list
{"name":"docker.io/library/busybox","tags":["latest"]}
# curl -kXGET https://[registry]:5000/v2/docker.io/library/busybox/manifests/latest
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"docker.io/library/busybox","Revision":"sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101"}}]}
# docker restart local-registry
# docker push [registry]:5000/docker.io/library/busybox
The push refers to a repository [[registry]:5000/docker.io/library/busybox]
e88b3f82283b: Pushed
latest: digest: sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101 size: 505 |
The issue also exists. |
This issue cased by the registry cache. I disable the registry cache to solve this problem. And it has worked. |
Closing as wildly outdated. Please open a new issue after testing on the latest available release. |
Hi,
I am trying to cleanup a 2.5.0 registry which is getting its main disk full.
I deleted some tags, run the garbage cleanup cron job, disk was freeed, but when I try to repush an image that was deleted, I hit the following error "Layer already exists".
I have also originally posted here:
docker-archive/docker-registry#988
While I think this is the appropriate place to mention the issue.
Can someone from Docker confirm the issue?
This is really an annoying bug, because it prevents the real "Continuous Integration" to happen.
The text was updated successfully, but these errors were encountered: