document support for --enable-helm flag for kustomize

[djfinnoy]

Summary

I would like to utilize the --enable-helm feature of Kustomize (v4.1.0), without writing my own custom plugin.

Motivation

Kustomize provides a neat way to handle last mile modification of Helm charts. At the moment, I believe the textbook solution for this is to make a custom plugin utilizing Helm's postrender feature. I think supporting Kustomize's helmChart functionality may be a better option for some users. Implementing support for this in ArgoCD would allow us to easily leverage both tools.

Proposal

spec.source.kustomize could include a new option, which incorporates --enable-helm under the hood when attempting to render the source manifests. Eg:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  source:
    repoURL: https://github.com/my-org/my-app
    targetRevision: main
    path: .

    kustomize:
      enableHelm: true   # This appends --enable-helm to the kustomize build command

[lfdominguez]

Hi, I'm using --enable-helm already, you must customize in argocd-cm ConfigMap with key kustomize.buildOptions, only add: --enable-helm, currently I'm using --load-restrictor LoadRestrictionsNone --enable-helm to allow refer to others common components outside kustomize folder.

[djfinnoy]

@lfdominguez thanks for the tip, that's definitely better than a custom plugin.

[srispl]

It looks like --enable-helm at argo-cm level will enable for all apps. Though the flag wont impact for apps not using helm charts, using at platform level impacts all clusters. Creating custom plugin will definitely helps if we need separate the flag only for specific applications.

What you need to do is

1. Create custom plugin in argocd-cm (This is one-time creation)

data:
  configManagementPlugins: |
    - name: kustomize-build-with-helm
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]

2. Provide the custom plugin in Application (Config required at app level)

  project: myproject
  source:
    #Custom plugin to enable helm
    plugin:
      name: kustomize-build-with-helm

[djfinnoy]

It looks like --enable-helm at argo-cm level will enable for all apps. Though the flag wont impact for apps not using helm charts, using at platform level impacts all clusters. Creating custom plugin will definitely helps if we need separate the flag only for specific applications.

Indeed, but as I said in the original post; I would like to do this without making a custom plugin. The reason is that if I want to use any of the built-in kustomize configuration options along with --enable-helm, I would have to code them into the custom plugin. This is why I feel like including the enable-helm option among the built-in kustomize config would be a better approach for most ArgoCD users who need it.

[srispl]

Second that @djfinnoy

[nhuray]

@lfdominguez I tried your suggestion but unfortunately it looks I have an issue to render a kustomized helm chart (ArgoCD 2.2.5).

Here is my configuration:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: hello-nodejs
    version: 0.1.7
    repo: https://github.com/IvanovOleg/charts/main/

The configmap:

apiVersion: v1
data:
  kustomize.buildOptions: --enable-alpha-plugins --enable-helm --load-restrictor LoadRestrictionsNone
...
kind: ConfigMap
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd

But the app is not created:

Any thoughts ?

[lfdominguez]

You must set a real helm repository, not that URL

[nhuray]

@lfdominguez

When I run kustomize in local with that config, the helm chart is rendered and patched :

So why it would be different in ArgoCD ?

[nhuray]

I found the issue. It wasn't related to the url or a misconfiguration for kustomize but simply because the Application I'm creating so far are based on Helm charts. So I had something like that:

project: dev
source:
  repoURL: 'git@github.com:estateably/deployments.git'
  path: services/test
  targetRevision: dev
  helm:
    valueFiles:
      - values.yaml
    releaseName: test
destination:
  server: 'https://myserver'
  namespace: test

Removing the helm configuration fixed the issue. Thanks @lfdominguez for your help !

[lfdominguez]

repoURL: '

Sorry i learn now that you can use the raw path of the url to get the chart... thanks

[igorcezar]

Any updates on this?

It looks like --enable-helm at argo-cm level will enable for all apps. Though the flag wont impact for apps not using helm charts, using at platform level impacts all clusters. Creating custom plugin will definitely helps if we need separate the flag only for specific applications.

What you need to do is

1. Create custom plugin in argocd-cm (This is one-time creation)

data:
  configManagementPlugins: |
    - name: kustomize-build-with-helm
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]

2. Provide the custom plugin in Application (Config required at app level)

  project: myproject
  source:
    #Custom plugin to enable helm
    plugin:
      name: kustomize-build-with-helm

It will impact performance for all clusters if enabled on kustomize.buildOptions, even if most applications don't use it?
From what I see in local tests it doesn't impact at all when building apps that don't use it, so I don't see what would be the impact for the clusters.

[lodotek]

Any updates here?

[trexx]

I'm using a private helm repo with a private CA. Enabling the option via kustomize.buildOptions causes x509 unknown authority errors as it seems ArgoCD isn't yet able to pass the --ca-file parameter when kustomize performs a helm pull.

[crenshaw-dev]

ArgoCD isn't yet able to pass the --ca-file parameter when kustomize performs a helm pull

@trexx the helm pull would be invoked by Kustomize rather than Argo CD. Does Kustomize provide a way to pass through a CA file when --enable-helm is used? If so, you could also set it in argocd-cm.

[crenshaw-dev]

I'm not understanding the reasons given why the argocd-cm method isn't sufficient. Is there some reason this needs to be configurable at the Application level rather than at the global level?

[trexx]

ArgoCD isn't yet able to pass the --ca-file parameter when kustomize performs a helm pull

@trexx the helm pull would be invoked by Kustomize rather than Argo CD. Does Kustomize provide a way to pass through a CA file when --enable-helm is used? If so, you could also set it in argocd-cm.

Kustomize has a --helm-command parameter that allows customising the "helm pull" command. There I can use --ca-file and hardcode a path to a certificate in /app/config/tls. My first attempt failed as the helm parameters were interpreted by Kustomize, I guess I need to escape some quotes somewhere.

Alternatively I was also thinking about editing the containers ca bundle or overriding it via SSL_CERT_DIR envvar. That maybe easier done in a CMP via a shell. That way, I can still follow normal ArgoCD procedures and add additional CAs via the documented procedures without needing to change buildoptions or the like.

      generate:
        command: ["/bin/sh", "-c"]
        args: [ "SSL_CERT_DIR=/app/config/tls/ kustomize build --enable-helm" ]

Go accepts a semicolon delimited list of paths in SSL_CERT_DIR.

[djfinnoy]

@crenshaw-dev I think the argocd-cm method is sufficient; it should probably be mentioned in the docs under a sub-header on combining kustomize and helm (the --enable-helm feature is well hidden in the kustomize docs).

Quote from the kustomize:

The flag --enable-helm exists to have the user acknowledge that kustomize is running an external program as part of the build step.

Making this acknowledgement for all kustomize apps should be fine, I don't see any compelling reason why ArgoCD users should need the ability to do this on the application level rather than the global level.

@trexx This challenge is more interesting. You would require both the --enable-helm and --ca-file flags to be built into ArgoCD's kustomize source if you wanted to do something other than using a custom plugin. Personally, I think that your use case warrants a custom plugin; I think the problem is too niche to warrant changes to ArgoCD itself.

Happy for this issue to be closed if a task on updating the docs is registered somewhere else.

[crenshaw-dev]

@djfinnoy I'll just turn this issue into a docs issue. :-D

[rufreakde]

Hi all,
I am new to the kustomize scene and have only 1 use case currently because the loki helm chart we use need some patching so I can replace the ConfigMap with a custom Secret. From this Issue and from the Documentation I was able to come so far:

My setup so far: (argocd-cm is configured with kustomize.buildOptions: --enable-helm as described by @lfdominguez )

# folder structure
argoapplication-loki-helm.yaml
patches/loki-read.yaml
patches/loki-write.yaml
kustomization.yaml

So now I am kind of hung up on the last step how to patch the ArgoApplication? (which is a helm install)

# kustomization (unfinished)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
annotations:
  argocd.argoproj.io/sync-wave: "150"
patches:
  - path: ./patches/loki-read.yaml
    target:
      name: loki-read
      kind: StatefulSet
  - path: ./patches/loki-write.yaml
    target:
      name: loki-write
      kind: StatefulSet

an example patch:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki-read
spec:
  template:
    spec:
      volumes:
      - name: config
        secret:
          secretName: loki-custom

and the App:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: loki-application
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "50"
spec:
  project: default
  source:

    chart: loki
    repoURL: https://grafana.github.io/helm-charts
    targetRevision: 3.6.0
    helm:
      releaseName: TEST
      values: |
        loki:
        ...
   
  destination:
    server: https://kubernetes.default.svc
    namespace: TEST
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - Validate=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - ApplyOutOfSync=true

So far my limited knowledge and googling does not make the jump to connect kustomize with the argoCD app. Does anyone know what I am missing for the final stretch?

[chary1112004]

@rufreakde we have similar case as your case. We need customize configmap in bitnami mongodb helm chart.
Did you find out the complete one? In case yes, could you please share? Thanks!

[rufreakde]

@rufreakde we have similar case as your case. We need customize configmap in bitnami mongodb helm chart. Did you find out the complete one? In case yes, could you please share? Thanks!

@chary1112004 this should be a full working example dnk why documentation of a full blown example is so hard to find and if so only in bits and pieces one has to put together.

How to overwrite the loki secret of the loki helm chart:

├── loki-application.yaml
├── loki-custom-config-secret.yaml
├── loki-kustomization
│   ├── kustomization.yaml
│   ├── overlay
│   │   ├── loki-read.yaml
│   │   └── loki-write.yaml
│   └── values.yaml

argocd-cm.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-10"
data:
  ...
  configManagementPlugins: |
    - name: kustomize-build-with-helm
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]

loki-application.yaml

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: loki-application
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "60"
spec:
  project: default
  source:
    # Custom plugin to enable helm
    plugin:
      name: kustomize-build-with-helm
    repoURL: <URL>.git
    targetRevision: main
    path: loki-kustomization
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - Validate=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - ApplyOutOfSync=true

kustomization.yaml

---
helmCharts:
- name: loki
  version: 3.6.0
  repo: https://grafana.github.io/helm-charts
  releaseName: monitoring
  namespace: monitoring
  valuesFile: values.yaml
  valuesMerge: merge
  includeCRDs: true

patchesStrategicMerge:
- overlay/loki-read.yaml
- overlay/loki-write.yaml

overlay example loki-read.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki-read
spec:
  template:
    spec:
      volumes:
        - name: config
          configMap:
            $patch: delete
          secret:
            secretName: loki-custom

[chary1112004]

@rufreakde thank you a lot for sharing!

[rufreakde]

@crenshaw-dev this is again now an issue since starting with argocd version 2.6 or later the argocd-cm plugin is deprecated.
And there is no declarative way to enable a custom plugin anymore. (at least without a custom image)

Why is there no way to declaratively define a plugin. We do not want to patch sidecars is there no way to have a configmap as before?
https://argo-cd.readthedocs.io/en/release-2.7/operator-manual/config-management-plugins/#register-the-plugin-sidecar

It is kind of a problem to have argocd not allowing a declarative configuration in this case.

[crenshaw-dev]

@rufreakde sidecar CMPs can be defined declaratively. You can define the sidecar patch and config via Kustomize overlays or via values in the Argo Helm chart, or a variety of other ways.

There is a usability cost. I'm going to open an enhancement request to describe how we can make installing a sidecar CMP almost as easy as an argocd-cm plugin.

[crenshaw-dev]

#15006

[rufreakde]

Thanks ArgoCD is the only thing we do not install via helm or kustomize on our side that is why we do not have kustomize or something. Instead what we have is an ArgoCD managed application that just deploys/updates configs via argoCD itself.
E.g. the argocd-cm or other files like some secrets etc. and apply the usual command on updates that is also in the release readme.

Thanks for opening this enhancement. Just now we also realised that the plugin section still works in àrgocd-cm on our side when we updated to version 2.7.11 even though the docs shown something else?

[crenshaw-dev]

@rufreakde yep we bumped the removal a couple times. argocd-cm plugins are finally gone in 2.8.

[gajus]

@rufreakde yep we bumped the removal a couple times. argocd-cm plugins are finally gone in 2.8.

Can someone provide an example how to migrate:

configManagementPlugins: |
  - name: kustomized-helm
    init:
      command: ["/bin/sh", "-c"]
      args: ["helm dependency build || true"]
    generate:
      command: ["/bin/sh", "-c"]
      args: ["helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds $ARGOCD_ENV_HELM_ARGS > all.yaml && kustomize build"]

To whatever is supported at the moment?

[crenshaw-dev]

@gajus there are migration docs here: https://argo-cd.readthedocs.io/en/latest/operator-manual/config-management-plugins/#migrating-from-argocd-cm-plugins