feat(appset): Implement Plugin Generator

[binboum]

ApplicationSet generators are useful for modeling templates using external data sources to deploy applications.

Today, generators have been developed based on the needs of the community, and when a new need arises, it's necessary to modify the Appset codebase.

The proposal here is to have a "plugin" generator that would allow extending the codebase according to specific needs, without having to modify it directly.

It's inspired by the Plugin functionality of Argo Workflows

PR includes:

• Proposal
• Generator Plugin ( Matrix, Merge, Webhook are included )
• Internal library HTTP for generator
• First draft of documentation

Labs :

• https://github.com/binboum/argo-test/tree/main/apps ( standalone, merge, matrix )
• https://github.com/binboum/argo-test/tree/main/infra/plugin ( feature branch plugin POC )
• https://github.com/binboum/argo-test/blob/main/.github/workflows/back.yml#L36 ( Pipeline for update feature branch digest )
• https://github.com/binboum/argo-test/tree/main/bench ( benchmark script for create dynamic feature branch ).

Discussions : https://cloud-native.slack.com/archives/C01TSERG0KZ/p1677240673615059 @crenshaw-dev @scrocquesel

Related :

#12429
#12368

Credits @binboum @scrocquesel

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.

Please see Contribution FAQs if you have questions about your pull-request.

[codecov]

Codecov Report

Patch coverage: 68.26% and project coverage change: +0.20 🎉

Comparison is base (129892b) 49.27% compared to head (1f7ff87) 49.48%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #13017      +/-   ##
==========================================
+ Coverage   49.27%   49.48%   +0.20%     
==========================================
  Files         251      256       +5     
  Lines       43541    43847     +306     
==========================================
+ Hits        21455    21697     +242     
- Misses      19951    19988      +37     
- Partials     2135     2162      +27     

Impacted Files	Coverage Δ
...cationset/services/internal/http/client_options.go	0.00% <0.00%> (ø)
.../apis/application/v1alpha1/applicationset_types.go	28.23% <0.00%> (-0.34%)	⬇️
applicationset/generators/merge.go	53.20% <50.00%> (-0.05%)	⬇️
applicationset/utils/utils.go	79.00% <50.00%> (+7.98%)	⬆️
applicationset/services/plugin/plugin_service.go	60.00% <60.00%> (ø)
applicationset/services/internal/http/client.go	62.50% <62.50%> (ø)
controller/state.go	67.98% <66.66%> (ø)
applicationset/generators/plugin.go	75.52% <75.52%> (ø)
applicationset/generators/matrix.go	77.86% <100.00%> (+0.36%)	⬆️
applicationset/services/plugin/utils.go	100.00% <100.00%> (ø)
... and 2 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[crenshaw-dev]

@binboum Intuit has an internal hackathon 3 weeks from now where I plan to build a POC for this on an internal use case. Looking forward to helping move this PR forward!

[binboum]

@binboum Intuit has an internal hackathon 3 weeks from now where I plan to build a POC for this on an internal use case. Looking forward to helping move this PR forward!

Hello @crenshaw-dev

On our side, the PR is functional technically, we kept it in draft because we wanted to have a feedback on the architecture discussed with the proposal, but in fact, it is functional and tested.

What we have to push now is the documentation, but we were waiting for a return from you :)

[crenshaw-dev]

@binboum we're going to start a PoC Monday. :-)

[crenshaw-dev]

End of day 1: already have a plugin set up querying an internal API to generate our Argo Rollouts deployment apps. It was super easy. I expect to be able to open a PR against this one with changes we've discussed above, to save y'all some coding. :-)

[crenshaw-dev]

Code as promised: binboum#2

[crenshaw-dev]

Do we / should we hot-load plugin configmap changes?

[scrocquesel]

Do we / should we hot-load plugin configmap changes?

AFAIR, yes, the http client is recreated at reconcile time from the config map which is cached with an informer. @binboum do you confirm that ?

Regarding "should we", secrets/secret refs/appset are hot reloaded so I guess admins expect the conf to be hot reloaded and so does the plugin config map.

[binboum]

Do we / should we hot-load plugin configmap changes?

AFAIR, yes, the http client is recreated at reconcile time from the config map which is cached with an informer. @binboum do you confirm that ?

Regarding "should we", secrets/secret refs/appset are hot reloaded so I guess admins expect the conf to be hot reloaded and so does the plugin config map.

Yes I confirm, and it is tested.

[crenshaw-dev]

@binboum I made a few more changes on my PR. Can you review/merge? binboum#2