Git tags not resolving to the right commits

[PaulSonOfLars]

Describe the bug

Pinning to a git tag uses the wrong commit.

To Reproduce

An example app to reproduce this is available here: https://github.com/PaulSonOfLars/test-argo-app

Note that the repo has two tags:

• 2024-09-22-test, against commit 4650a611850c03600e423d8385ab237308a8ff05
• 2024-09-22-test-2, against commit 6650b7ff386c9cb8a2e199ae1fd78571aa3ac336

The app is pinned to 2024-09-22-test, so SHOULD be using commit 4650.... However, it is not - see below;

Expected behavior

ArgoCD should use the commit that the tag is pinned to (4650...). However, it is using 6650... instead.

Screenshots

See git history below, with commits etc.

Version

v2.12.3+6b9cd82

Logs

Have enabled debug logs, but can't see anything of note.

My initial reaction is that this may be an issue to do #17566; somehow the tags above are being treated as semver, when they shouldn't be. This then leads to the tags being updated to later commits, even though we expect them to be fixed.
This would explain why ArgoCD uses the second tag, rather than HEAD.

Does that sound about right, or am I barking up the wrong tree?

[anandf]

This breaking change has been introduced between v2.11.0-rc3 and v2.11.0.
I tested this regression with the git repo -> https://github.com/anandf/argocd-example-apps and revision v0.1.0 which is an annotated tag.

argocd app create guestbook --core --repo https://github.com/anandf/argocd-example-apps --path kustomize-guestbook --revision v0.1.0 --dest-namespace guestbook --dest-server https://kubernetes.default.svc --sync-policy automated --self-heal --sync-option CreateNamespace=true

Broken in v2.11.0

v0.1.0 (256a701) -> is the annotated tag commit id pointing to anandf/argocd-example-apps@256a701
Last working version in v2.11.0-rc3

v0.1.0 (d7927a2) -> is the actual commit id pointing to a valid url anandf/argocd-example-apps@d7927a2

I am suspecting this commit to have caused this regression. da6c2e9

[aenshin-pp]

@PaulSonOfLars here is good article with explanation of differences between lightweight and annotated tags:
https://rem.co/blog/2015/02/12/git-the-difference-between-lightweight-and-annotated-tags/

Here is the same issue which happened before to Argo CD:
#4231

We also experience this issue right now in our org after Argo CD update.

[PaulSonOfLars]

@aenshin-pp thanks for the reference! Sadly, I don't think this is to do with lightweight/annotated tags - I did come across the PR you've linked while I was investigating, and double checked that I am definitely using lightweight tags for all purposes here.

I can confirm this because if I pin to a tag 2024-09-22-test, it will correctly target the expected commit. This will remain as expected even if i push new commits.
But as soon as I create a new tag 2024-09-22-test2 (and DONT change the pin), the app will instead still start tracking the new commit referenced by the new tag (even though the app still pins to 2024-09-22-test)

I opened up #20096 and wrote some tests to confirm my assumptions, and can confirm the issue; argocd seems to be interpreting genuine non-semver git tags as semver constraints, which are then being smudged into using "newer versions".

[PaulSonOfLars]

I am suspecting this commit to have caused this regression. da6c2e9

@anandf that seems surprising, given that commit doesnt seem to have anything to do with tags or git, and argocd didnt have semver resolution at the time - that was a feature introduced in v12... Hopefully i havent completely misunderstood the issue here.

[aenshin-pp]

@PaulSonOfLars got it. Your issue sounds more complicated than mine :)
Mine is here: #20082 and it is the same as old one #4231 .

The issue relates to annotated tags and easy to reproduce which @anandf has confirmed.

[blakepettersson]

@anandf for me it seems to be the other way round 🤔:

v2.11.0-rc3 (256a701):

v2.11.0 (d7927a2):

[blakepettersson]

Oh interesting, when running v2.11.0-rc3 again, I get this:

So I guess what this says is that

1. this error predates v2.11.0-rc3
2. this doesn't always happen

[blakepettersson]

Hmm, I noticed that with rc3 that it starts with d7927a2, but after a while the sync status starts to show 256a701.

[PaulSonOfLars]

@blakepettersson Thanks for looking into this! Unsure if this was made clear from the above conversation, but just to avoid any confusion, as it seems youre investigating the wrong repro -
The issues raised by @anandf and @aenshin-pp are related to lightweight vs annotated tags, and are a different issue to the one I'm raising here, which is regarding unexpected semver resolution of git tags.

The repro I've provided in the issue description is only using lightweight tags; adding a new tag starting with the same number will then "bump" to the new commit, while showing the old tag, because tags are being treated as semver constraints rather than an actual tag

[blakepettersson]

@PaulSonOfLars I'm aware that this issue is different from yours, I just wanted to give this a quick spin.

I'll take a look at your issue in a bit - preliminarily my thinking is that your PR should be merged as-is, and then a subsequent PR would try to consolidate the semver resolution between Helm/Git/(and soon OCI). Will confirm my initial thinking in a bit 😄

[PaulSonOfLars]

Ok great - thanks for clarifying! Just wanted to make sure :)

FWIW - I agree with breaking it down into two steps, and have some code ready for that second PR already. Since I'm already familiar with that corner of the code, I figured I might as well!

[blakepettersson]

@anandf @aenshin-pp I git bisected this, and it seems like the actual culprit is #17874, which got introduced in 2.11-rc3.

cc @alexmt

[ryanisnan]

We just got burned by this.

[blakepettersson]

@ryanisnan by annotated tags or faulty semver resolution? (nb: we should move the annotated tags discussion to #20082 to reduce the confusion)

[blakepettersson]

@PaulSonOfLars's PR #20096 fixed this issue. For the other issue we can carry on in #20082.

[ryanisnan]

Thanks for the reply @blakepettersson. I'm not too sure which specific bug we ran into, but we suffered the same effect of argo not resolving a git tag to the correct commit.

The upgrade caused a number of our helm charts to roll back to a seemingly random commit from a year ago :(

[blakepettersson]

@ryanisnan could you provide some details, e.g your Application spec? Are you using semver resolution?

[modulo1982]

We are experiencing this issue on 2.12. Will this fix be backported? This currently causes unexpected behaviour.

[blakepettersson]

@modulo1982 which issue are you experiencing? There are two issues talked about in this thread - the semver fix will be backported to 2.12.

[modulo1982]

We are running into the semver issue we are running into. Good to hear it will also hit 2.12. And also found the pull request for 2.12 so will follow that. Thanks!