-
Notifications
You must be signed in to change notification settings - Fork 498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Athens leaks defunct zombie git processes #1155
Comments
@bradfitz Athens does not use
In both cases, we run Otherwise, I imagine this might be a bug in go modules itself? |
Also iirc Athens v0.2.0 is built with go version 1.11.x (I think 1.11.4 but not at my machine to confirm), in case that helps |
@arschles Athens v0.2.0 uses If I'm looking at the code from 1.11 correctly, running git commands for modules comes from here which seems to use Hopefully I'm not looking at the right code because otherwise I'm not sure where the processes are being abandoned. @bradfitz is there any way to see the actual command that ran those abandoned processes? Thanks |
Actually I noticed that athens-proxy is pid 1 above. I suspect this is just a packaging issue in your container. You should probably wrap athens-proxy in https://github.com/krallin/tini (see golang/go#23705 where Go discovered this too for some of our services) |
Disclaimer: I'm using this issue as a learning moment because it's the first time I hear about Docker/GKE not cleaning up properly finished processes. So thank you very much for taking the time to report the issue and help out :) As for your last comment,
Do you mean we should just include Here is the Dockerfile we use for reference: https://github.com/gomods/athens/blob/master/cmd/proxy/Dockerfile According to Tini's author in What is the advantage of Tini?, if the code knows not to leave defunct processes then you don't need Tini. Tini is mainly for running processes that you don't have control over like Jenkins. In our case we do have control over both Athens and Go. So my thinking is that Tini would work but it would not solve the underlying solution of some bad code somewhere not properly waiting on child processes. The only possibility I see is that Furthermore, I saw the following in the
Can you just pass the Which makes me think that Lastly, the issue you linked seemed to be a GKE problem here: golang/go#23705 (comment) Could this mean that GKE might have this issue fixed and all you need to do is update the cluster? Lastly, I'm obviously very happy to include It also seems like Thanks again! |
Hmm i think i saw those defunct git processes at some point with 0.2, havent seen that happening since i have updated to 0.3. It was maybe related to this census-instrumentation/opencensus-go#1046. |
Nevermind i just checked and can see 33 zombie git processes with |
What do folks think about this problem? @bradfitz did you happen to update GKE and see if that fixed the problem? Otherwise, can we give you a custom image, which executes Athens with |
I personally have done something similar to https://github.com/ramr/go-reaper in my own containers. |
@arschles I don't have any issues with tini. It makes a lot of sense since Athens already has dependencies from alpine such as vcs and openssh packages. |
@atomi that's all good! I'm really glad you brought up your go-reaper project 😄, I'll go check it out sometime. Cheers! |
@arschles It's not my project. I've just used it in the past before It' was an interesting solution at the time. Anyway, Cheers. |
Describe the bug
I've had an Athens (0.2) pod running in Kubernetes since 24 Jan 2019. It's leaked 3,675 zombie processes since then.
Error Message
To Reproduce
Run Athens for a bit.
Expected behavior
Processes are waited upon and don't leave zombies.
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: