Clarify Argo CD version support/alignment

[nbarrientos]

Is your feature request related to a problem?

No response

Related helm chart

argo-cd

Describe the solution you'd like

This is a follow up of the discussion that took place in #2705.

Argo CD supports the last 3 minor versions with security and bug fixes. For instance, recently GHSA-9766-5277-j5hr got patched in 2.9.15, 2.10.10 and 2.11.1.

However, it's not very clear if argo-helm follows that release cycle. Hence, could it please be clarified if for instance versions of the Helm chart newer than 6.7.x (e.x. 6.10.0) support Argo CD v2.10.x? For instance since this commit they might not but it's not clear to me what's the lower bound of supported Argo CD versions when you bump Chart.yaml/appVersion. In short, how can I tell what's the minimum Argo CD supported by a given version of the chart? Alternatively, do you only support the latest minor?

This question arises because in production environments using this chart to deploy Argo CD, admins typically cannot upgrade minors at short notice when there's a security fix. Actually, there might not even be a point in doing it if the minor being run is still supported as it's the case with 2.10.x today.

Thanks for your time!

Describe alternatives you've considered

No response

Additional context

No response

[rouke-broersma]

From the readme:

Argo Helm is a collection of community maintained charts for https://argoproj.github.io/ projects.

There is currently no strategy for backporting as far as I can tell so the charts are roll-forward only. As such a chart version can only be assume to be valid for the version of argo-cd that is shipped with that chart version. Anything else, you're on your own.

[nbarrientos]

Hi! Thanks for your input. Personally I don't see any relationship between the fact that the charts are community maintained and the issue discussed here. If the community wanted so there could be an alignment with Argo CD's release cycle.

Is this your personal opinion or the position of the maintainers?

[rouke-broersma]

The relation between these charts being community maintained and your question as to argocd version support per chart version is that since many upstream projects (argocd support matrix is surprisingly large tbh) don't even have the capacity to backport fixes to their applications, it is safe to assume a community managed chart will definitely not have backport fixes nor backwards compatibility with older application versions.

I am not a maintainer of argo-helm thus I cannot speak to their official position. However I can make an educated guess based on the available information, which is what I did. Feel free to wait for an official reply however, i'll be pleasantly surprised if I'm incorrect.

[pdrastil]

Hi @nbarrientos as stated correctly above we don't have strategy for back-porting manifest changes and currently latest chart versions corresponds with latest supported stable release. In general manifests and CRDs should be backwards compatible in same major version of the chart (6.x, 7.x, etc.). To get CVEs for older versions of Argo CD you can pin chart version & image tag in values and override these if necessary like this:

global:
  image:
    repository: quay.io/argoproj/argocd
    tag: v2.10.11  # providing this overrides anything set in appVersion

In my personal opinion back-porting changes in chart would be challenging at least for these reasons:

• versioning - How to deal with breaking changes in values.yaml in relation to upstream tag? There is no simple way to correlate chart version with appVersion as both are independent artifacts.
• feature support - How to back-port new features from contributors to older manifests that might be different (situation where newer chart drops support for deprecated options but old chart still have them).
• team size - Amount of people that maintains charts in their free time is relatively small. I don't believe we would be able to effectively support multiple releases even if above would be somehow solved.

[nbarrientos]

Hi @pdrastil

Thanks for your message!

Not sure your pinning approach would have worked in this case. I believe that pinning the chart wouldn't have been an option because fixing this precise CVE (which is a good example to illustrate the issue!) is not just an Argo CD bump, support from the configuration management system of choice is necessary, too. In the case of the Helm chart this support (default Redis password setup) only came in the latest version available. I'll try to describe the problematic with an example:

When the CVE was published we were running Argo CD v2.10.9 driven by chart version v6.7.18. If I understand things correctly, our options to fix the security issue would have been:

1. Upgrade Argo CD to v2.11.1 using chart version v6.10.0. This way the first version available of the accompanying chart necessary to patch the security issue would be guaranteed to be compatible with the software. However, this approach wouldn't have been adequate for us because the risk of jumping into a new minor which could contain new functionalities (from v2.10.x to v2.11.x) is much higher than a patch version increase and it would require extensive validation that typically takes days. Depending on the severity of the vulnerability being fixed this may be acceptable or not. In the worst case scenario (let's say non-authenticated RCE) running unpatched systems for say two weeks is not acceptable and I would not pass the blessing of the security teams. The validation of an upgrade containing not just the security fix but new features and bug fixes, even though being backwards compatible, is totally necessary. Not just by the Argo CD administrators themselves, but by all the application owners using Argo CD to deploy their workloads. For instance, in certain periods of intense LHC activity it'd be very problematic if breaking changes landed in production due to improper validation in staging leading to, well, I imagine we all know very well about the consequences 😁

2. Upgrade Argo CD to the next patch version with the security fix (v2.10.10), (forcibly) bump the chart to v6.10.0 (which might or might not be compatible with this Argo CD version) and, using the same phraseology that has been used previously, "be on our own". This is also not a very smart approach as we'd be driving Argo CD with a chart version that hasn't been tested for it. The longer the distance between the version of Argo CD the chart is meant for and the version of the software (somebody could be even running v2.9.15!) the more likely that regressions and incompatibilities may arise. It'd a bit like stepping into "undefined behaviour" land.

Please let me know if there's any other approach you could think of.

That said, the aim of this ticket was not to ask for a change in the support policy but simply to, as the title states, clarify what's on offer. It's entirely understandable of course if due to whatever reason (personpower, complexity, lack of need) your policy is to only officially support the greatest and latest. I understand that maintaining different flavors requires effort and maybe those who maintain the chart don't have the same stability requirements as others might have.

The main goal for me here is to make an informed choice at the time of selecting the tooling to know what to expect and to determine if the mechanics that relate to support and release cycles match our production reality.

The ticket can be closed. Maybe it'd be a good idea to add a little summary about this to the README so others with the same concerns can easily find the answer instead of having to navigate through the issues and dig it out from here. I can submit a PR myself if you want, just please let me know.

Thanks again!

[rdelpret]

I want to echo @nbarrientos sentiment here.

The fix for GHSA-9766-5277-j5hr introduces new manifests as part of the mitigation so if I am on a version of the helm chart that uses 2.8.x and I follow the guidance to pin .Values.global.image.tag to 2.8.19 it is my understanding that I have not actually mitigated the CVE. Luckily without the new manifests Argo CD "fails open" but as I understand it I would still be exposed to the CVE if a Redis password does not get set.

Unfortunately, Argo CD is one of those applications where the manifests that deploy it are very tightly coupled to the version of the code that they deploy. There was no documentation in the CVE or upgrade notes that say these manifests were being added that I could find. To find them I needed to look at the git diff from 2.8.18 and 2.8.19.

So it seems that back ported security patches that happen in argoproj/argo-cd are only valid for folks deploying with Kustomize. If this continues to be the case I think there should be some disclaimer for users of the helm chart which I believe is @nbarrientos point.

So my options at this point are:

1. Switch to Kustomize as ArgoCD does back port these manifest changes for CVEs (time consuming).
2. Upgrade 4 minor versions to get the CVE fix (time consuming).
3. Fork the helm chart and attempt to hack together a version compatible with 2.8.19 (risky and time consuming).

I understand the predicament for tracking Argo CD's releases though. If there is a bug in the helm chart or a needed feature like exposing some new feature in a new version of k8s you would have to wait for the next release of Argo CD to address it. Whereas Kustomize has the flexibility for the end use to pretty much do anything they need without the constraints of a values file. On the other hand, it gives you the ability to go back to the 2.8.18 helm chart and cut a 2.8.19 version. I wouldn't necessarily miss values file features I didn't have from latest if I was on 2.8.18.

Maybe this repo could cut special tags for security fixes only? Which probably means go find the version of argo-helm that ran 2.8.18 and cut a version of that helm chart with a special security fix tag. Or cut tags when Argo CD releases to anchor it to an argo CD release and someone in the community can attempt a patch version from that tag if needed (CVE for example).

I believe Argo CD commits to back porting security fixes to 3 previous minor versions and while I can't speak to how much work this would be, if you only supported CVE patches it would likely be less work than supporting all patch versions on previous minor versions.

[nbarrientos]

Yup, thanks @rdelpret for taking the time to drop your thoughts over here. I totally agree with all your statements.

[rouke-broersma]

@rdelpret @nbarrientos I don't see any reason why this could not be implemented, as long as someone who is interested in this is willing to put in the effort to make this happen and to supply the backports.

I don't think an official backport commitment is fair to expect from a community maintained chart, but I'm sure the infrastructure could be added to make it possible to do backports on-demand.

@pdrastil wdyt, if community members (users) themselves are responsible for contributing backports when they specifically require them, instead of introducing some kinds of 'official' backporting policy, would it be possible to introduce the CI infrastructure to support these backports through branches and tags?

The workflow I imagine would be relatively low-effort is:

1. User proposes specific backport on chart x for tag y
2. Repo member with write access when they have the time can create a branch from tag y for chart x
3. User contributes the backport with the backport branch as the target bumping the fix version
4. On merge pipeline releases this additional version (might already work with current pipelines?)

With this workflow the only additional time needed from repo members would be creating the branch to target the backport towards and reviewing the PR, and only if a user proposes a backport and requests the target branch. Any effort creating the backports would fall on the user that needs it.

@rdelpret @nbarrientos would this fit your needs?

[rdelpret]

I would be interested in how we would tag them in regards to the naming convention of the tags (like I said, could we anchor to an argo cd release so folks know its a backport for a CVE) but I think that is a totally reasonable compromise.

[rdelpret]

@rouke-broersma any update here?

[rouke-broersma]

@rouke-broersma any update here?

Not sure why you're tagging me. I gave a suggestion that as an outsider seems reasonably doable, while not putting a burden on the argocd maintainers.

It's up to someone that actually wants this (like yourself) to now put forward the proposal, build the necessary infrastructure, commit to supplying the backports and convince the argocd maintainers.

The whole point of my suggestion is that the legwork lays with the community that is asking for these backports. It's not going to be me. Apologies if it seemed from my comment that I would push the proposal myself, that was not my intention. Sorry if you've been waiting on me in this regard.

[rdelpret]

@rouke-broersma Sorry I thought you were a maintainer.

[rouke-broersma]

@rouke-broersma Sorry I thought you were a maintainer.

Sorry for giving that impression!