-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regression: Temporary git unavailability uninstalls applications #18212
Comments
cc: @todaywasawesome @similark PTAL |
Likely related to #14318 |
We faced something similar today and the impact was quite serious. Argo deleted a set of applications generated by 1 generator out of our 3 git generators.
We did not see a downtime on our version control provider's status page, but maybe there was a small glitch in the connection. Background:
EDIT:
Our first generator failed to generate applications
Second and third pass
So no error gets returned and the sync proceeds with the list generated by second and third generators |
We narrowed it down further and in our case the cause was the unavailability of the repo server.
|
It sounds like the fix would be to change the code to bail out when there's a connection error to the repo server. |
I think I'd be in favor of simply reverting #17062, unless you see an easy way to accomplish what the original author wanted without introducing the bug. |
I (#17062 author) will try implementing @crenshaw-dev ’s suggestion as I believe keeping that fix is important |
Makes sense! As I understand it, the worst case scenario after reverting is failure to create/update. The status quo seems to be accidental deletion, which seems much worse. If the choice is between the two, I think we have to go with reverting. But if we can have both, that's great! |
In my homelab, I've encoutered ocassional removal of argocd managed applications after upgrading to argocd 2.11. Upon further investigation, this always happens when my git server becomes temporarily unavailable.
I'm using https://github.com/argoproj-labs/argocd-autopilot to manage my homelab. In that setup, argocd-autopilot sets up a repository that defines an
ApplicationSet
that comes with twogit
generators:After inspecting the code around https://github.com/similark/argo-cd/blob/997f5c7da304efa209a32668d768966d75ee8c2e/applicationset/controllers/applicationset_controller.go#L128-L141 a
git blame
brought me to #17062 which I believe is causing this regression. Basically, the thinking in that PR is "if the number of generated projects is more than one, then it is save to take the resulting list as good.In the scenario that happens quite regulariliy, this isn't the case when argocd is unable to contact the the git repository for one, but not both git generators.
Version
Logs
The text was updated successfully, but these errors were encountered: