Support even ARGOCD in ec cli

[amichelotti]

I use ArgoCD to manage deploy synchronization of apps inside k8s. I found very useful because it helps to have production/development/,,, repositories aligned with their deploy. Has an essential GUI focused on applications (not as k8s dashboard) that can be understood also by people that dont know k8s. You can exactly know the status of the deployed application (iocs) in your cluster. It could be nice if 'ec' could generate yaml argocd application to launch iocs.

[gilesknap]

@amichelotti thanks for this. We have been looking for a GUI to give to beamline scientists and controls engineers for K8S and the standard dashboard is not ideal. We are also very keen not to make our own but use something that is tried and tested.

I've not looked at ArgoCD yet but will definitely do so.

@coretl this might be a good little project for someone. Maybe @marcelldls - you know ec quite well and it would be a good opportunity to get into K8S some more.

[coretl]

I know @garryod has looked at ArgoCD in the past, he probably has some thoughts...

[garryod]

Yeah, there's a ticket in with the cloud guys (@cjreyn et al) to test out ArgoCD on our cluster. I had primarily envisaged it being used to deploy & manage inflated Helm charts covering entire application stacks following the GitOps pattern. That's not to say it couldn't be used to deploy individual IOCs though, perhaps via a Kustomize'd manifest. If necessary you could even create a custom config management plugin that reads from the normal IOC deployment configuration

[gilesknap]

@amichelotti our data analysis and data acquisition groups are beginning to look at ArgoCD and it looks like they are likely to adopt it. If this happens then it becomes a lot more attractive for the controls team to use it for IOC deployment too.

As you are experienced in both ArgoCD and epics-containers I wonder if you could elaborate on how you would envisage using it with IOCs? Some questions come to mind (but I don't know that much about Argo so some may not make sense):

• Is having an ArgoCD repo and an epics-containers beamline repo redundant - should we have only one?
• Is the Argo CLI then a sufficient replacement for ec or would ec still be useful?
• I assume restarting an IOC would not require updating the repository but how about stopping and starting an IOC is this always going to go through the Argo repo?
  • If the latency is OK then this is not terrible because it keeps a record of every such event on the beamline I guess
• How about the deploy-local feature of ec? this is an extremely useful way to rapidly test IOC config before committing it - I don't think I would want to loose that function.

Let us know any other ideas you might have regarding how ArgoCD and epics-containers might integrate.

Many thanks.

[gilesknap]

BTW I like to live by Principal of Minimum Bespoke in other words don't write software that is already available as a common of the shelf item. Therefore if Argo could replace my home grown ec and beamline repositories then I'd be happy to get rid of those bespoke items I invented.

[gilesknap]

One item I am concerned about is that we use helm package in ec to do two things

• wrap up the contents of config into a ConfigMap
• embed the git tag version into the helm chart AppVersion

Therefore ec calls helm a couple of times per deployment - its not a straight helm update - I wonder if there is a way to get ArgoCD to do this?

[amichelotti]

Hi Giles/all,
Essentially, I put ArgoCD at the center of the workflow. Once you have a Kubernetes cluster and ArgoCD set up, everything is managed through ArgoCD. I have three main Helm charts:

1. An "IT" chart that installs MetalLB and filesystem support.
2. A "main backend services" chart that installs Epics-related services such as Kafka, MongoDB and Elasticsearch.
3. An "epics chart" that installs IOCs, services, and miscellaneous applications.

These three charts are root charts, meaning they contain and instantiate other charts. To deploy an IOC, a service, or a miscellaneous application for a specific beamline, I start with the main Git repository for the beamline (which is composed of subprojects). I then add the necessary lines in a values.yaml file to initialize the "epics chart." When this is pushed to the repository, ArgoCD (which can be set to autosync or not) creates the IOC, service, or application.

What I find very useful with this workflow is that it’s easy to teach, and even physicists who aren't very familiar with IT can easily deploy soft IOCs in Python (using DLS soft IOCs) to make their crazy algos. They can also check if the IOC or process is running properly through ArgoCD's simplified GUI interface, which can be used to remove, restart, and update applications, and view logs. For more sophisticated tasks, such as attaching to the console of an IOC and issuing commands, using ec or similar tools that wrap useful kubectl commands is recommended, especially for control system people.

In normal use (if no need to debug or tracing), the ArgoCD GUI is sufficient and user-friendly. By utilizing labels, namespaces, and other features, you can quickly find applications and IOCs. You can find the "epics chart" for ArgoCD (code-named epik8) here: epik8-chart. It is very simple and allows launching other charts available at epics-containers.

ArgoCD greatly facilitates integration with users who are more interested in the application layer (experiment data scientists) than in control systems. In summary, we use ec for our internal needs in particular to make debug and ArgoCD for everything else. This workflow ensures that everything is replicable—just use your "beamline" repository (with submodules), and ArgoCD can restart everything from scratch.

I have essentially a repository of 3 root charts that instantiates via ArgoCD other charts (public or custom). Those are generic and I rarely change. The beamline git contains 1 chart that is empty but its values.yaml instantiates the "epics chart (epik8s)" with a big yaml that contains all ioc,services and apps. It's also good to have just one point where change things and that things are traced by git.

I must mention that this workflow is still relatively new, and we have not yet moved it into production (the first facility is scheduled for the end of June). Currently, only a limited number of people are making modifications, but the ease of putting things into production could also pose challenges. I am aware that ArgoCD is becoming very popular within the INFN IT community.

I hope I have addressed your questions adequately. When our principal IT specialist returns from a conference, where he is presenting this work, he plans to write a comprehensive guide. This guide will include an Ansible playbook to install everything on a single node, allowing you to try out the full stack. In fact, this workflow is so simple that, combined with the "ibekized" IOCs, even physicists with no IT or software skills who need control for a small experiment could use such a single-node setup.

[gilesknap]

Thanks for a comprehensive response @amichelotti.

I still don't think I'm clear on the connection between your epik8s yaml and the beamline repo services. The helm charts described by the beamline repo services folders require a little tweaking by the ec tool in order to make them deployable and so I don't see how you can make a deployment just by editing the epiks8.yaml

(the tweaks are packaging the config folder and updating AppVersion)

I look forward to reading the guide when it is available, the whole approach sounds very encouraging.

[gilesknap]

I think I can answer my own question as that is what you started the discussion on. You wanted ec to support making its output appropriate for the epiks8 yaml.

As it happens I've just made a major refactor of ec that should allow dropping in different back ends - currently only supporting the original local podman/docker or K8S.

Can you describe here what that YAML output would look like? Would it be appropriate to have the k8s tool directly update the epiks8 repo?

[amichelotti]

Hi Giles,since I posted the idea, a little bit of things are changed and I automated a lot the way of deploying things. When I posted the idea was just to suggest wrapping the ec deploy in a ArgoCD application, so that I could manage it from ArgoCD gui. Now with the workflow that I'm following I just run a single kubectl apply -f <beamline.yaml> then ArgoCD with the provided chart creates recursively the rest. I dont use too much ec, even if I understand that it is useful to debug, build and test docker solutions. If I need to understand the details of my deployments I use OKD/k8s dashboards. Probably when I will have everything up I'll focus on ec again because there are some operations that are not supported by dashboard (as for instance attach on the running process) that are instead important to debug iocs.

The starting beamline.yaml has this form:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sparc-start-epik8-deploy
  namespace: argocd
  labels:
      deploy: sparc
spec:
  project: default
  source:
    repoURL: 'https://baltig.infn.it/lnf-da-control/epik8-sparc.git'
    path: deploy
    targetRevision: HEAD
    helm:
      parameters:
        - name: beamline
          value: sparc
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: sparc
  syncPolicy:
    automated:
      prune: true  # Optional: Automatically remove resources not specified in Helm chart
      selfHeal: true
    syncOptions:
      - CreateNamespace=true 
      - Prune=true

Then the chart https://baltig.infn.it/lnf-da-control/epik8-sparc.git cointains all the information to build a full system and it recursively call a list of charts.
The epik8-sparc beamline chart has a deploy directory that has a template dir with only one file (always the same):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: {{.Values.beamline}}-deploy-all
  namespace: argocd
  labels:
    deploy: {{ .Values.beamline }}
spec:
  project: default
  source:
    repoURL: 'https://baltig.infn.it/epics-containers/epik8-chart.git'
    path: .
    targetRevision: HEAD
    helm:
      values: |
          {{- toYaml .Values | nindent 12 }}

  destination:
    server: 'https://kubernetes.default.svc'
    {{- if .Values.namespace }}
    namespace: {{.Values.namespace}}

    {{- else }}
    namespace: {{.Values.beamline}}
    {{- end}}
  syncPolicy:
    automated:
      prune: true  # Optional: Automatically remove resources not specified in Helm chart
      selfHeal: true
    syncOptions:
      - CreateNamespace=true 
      - Prune=true

And a (big) values.yaml with the information about iocs/services/apps. Together with the beamline chart repo (in this case epik8-sparc) I also bound configurations of the services and iocs that are mainly ibekized. So that I've my beamline fully described and reproducible in one point
One other thing that I've different respect your docker images, I dont use predefined or shared folders for the configuration but I create a private PVC and the pod when starts initialize with the appropriate configuration directory of the beamline repo.

[gilesknap]

Thanks for a comprehensive description of an interesting setup.

So it sounds like you don't use our Beamline repo's or ec tool. Like I said that's something I would aspire to if ArgoCD can take their place

It does sound like there is manual editing of large yaml files required from your description. Is that the case?

[amichelotti]

Yes, at the moment I manual edit and people that want to add iocs,app have to clone the repo, add, test and commit. In my dream and big picture, in the future there will be a "super ibek or super ec" that from a high level description will render yaml for Argocd, ibek and auspicably also opis. This "high level description" should stay or should initialise some NRDB document store like mongo or even elastic search (that seems popular in the community), so that finally we can query and at last have a full database of the control.

[gilesknap]

OK well I'm now thinking of an extra backend to ec that makes the changes to the ArgoCD repo.

This would be a small step toward your goal. It would initially imply that the workflow using ec does not change but everything goes through ArgoCD so is fully audited. This still means using a repo to describe the beamline with a couple of short YAML files per IOC - but then we could think about replacing that part.

Do you really need your database to be the source code for your facility though? Is it not easier for developers to continue to use git repos and command line tools to make their changes to their controls systems. But then, because you have a source of truth - ArgoCD - you could generate the DB from that and build tooling around querying it.

[gilesknap]

(if you make a DB generate your control systems you have to think of everything that might be needed up front - in epics-containers we have the config folder deployed along with every IOC instance and developers are free to drop hand coded startup scripts and substitution files etc. in there. That way we can support things we have not thought of in ibek yet. I guess the DB route could take the same stance and store whole files in some tables - but that feels like a backward step)