Add git2go option to enable Azure DevOps

[phillebaba]

So this change has been a long time coming, and has had its ups and downs, but now lets hope that this will be the PR that fixes a major blocker for Azure DevOps users.

The git2go library works with Azure DevOps as it has greater support for more complex capabilities in the git wire protocol. A downside of the library is that it does not support shallow cloning of a git repository, meaning that it needs to fetch whole repository every time. Something that could in theory be expensive both in time and transfer data for very large repositories. As this is a very large downside the compromise solution is to implement a option in the GitRepository resource called gitImplementation to enable the use of the git2go library. This is turned off to default meaning that source-controller will default to using the usual go-git library people are familiar with.

Eventually when shallow cloning is supported in git2go we could consider fully migrating to only using git2go.
libgit2/libgit2#5254

There are a couple of things that have to be done before merging this:

• Know Host validation in v2
• Transport tests in v2
• Update flux2 docs explaining when to enable v2

@stefanprodan @hiddeco I would appreciate any early feedback in case there is some major design flaws.

Fixes #104

An additional selling point for this change is that git2go could in the future allow us to do self signed cert verification from memory as it has a certificate validation callback.

[phillebaba]

In general i think we need better ssh protocol validation for both v1 and v2 implementations

[stefanprodan]

We need to install libgit2 on the CI VM:

- name: Install libgit2
  run: sudo apt-get install -y libgit2-dev

[phillebaba]

The libgit2 version in apt is very old so we need to find another way to install the latest version.

[stefanprodan]

Maybe this one will work https://packages.debian.org/experimental/libgit2-1.0

[hiddeco]

That version is too old, see the table here. I am afraid we may have to build it ourselves.

[phillebaba]

I have been looking around and it seems like you are right @hiddeco. I really like yak shaving 😄

[hiddeco]

Stefan and I discussed another option: testing from within a Docker container, that would allow us to piggyback on the package that is available for Alpine images.

[relu]

Running the whole job in a container is fairly simple with GHA, removing the actions/setup-go@v2 step and adding this to the job section should do it:

container:
  image: golang:1.15-alpine

I think it's the best approach to keep the build and test environment as close as possible, it should probably be done for all the other components too.

[stefanprodan]

@relu if we use the container image than we can no longer cache go modules and all the builds will take 5+ minutes. We don't use CGO expect in here, so I don't see why would we give up caching everywhere.

[relu]

I don't think it will break caching, at least I remember that was not the case in a previous project I worked on, although it was npm caching that was used rather than go modules, I think the mechanism is the same though.

[stefanprodan]

Are you saying that GitHub mounts the Go cache from host into the golang:1.15-alpine container? If so that's news to me

[relu]

Well... technically the cache itself is not on the host, it's kept in GHA caches (limited to 5GB) and restored over the network (really fast on GitHub-operated runners). We use caching even on self-hosted runners and it works the same, though a bit slower as you can imagine. Let me put it to the test, gonna try it out in a separate PR.

[stefanprodan]

@relu we can't run the whole thing in a container, we need Kubernetes Kind, docker buildx, etc

[relu]

Yes, and we probably don't want to mess with dind. Failure log is here: https://github.com/fluxcd/source-controller/pull/216/checks?check_run_id=1458681336 (at least the caching works, so that's something).

[stefanprodan]

@relu I think we should move only the make test into a container.

[relu]

@stefanprodan I agree, I've tested multiple solutions for running everything inside a container and it doesn't look good. It is worth noting that it is possible to get all the tooling installed and caching works just fine, but it seems to fail for some reason during test execution and I think it's not worth digging deeper as long as we can simply just build and run a container for the test step alone. I can help out with that if you don't already have a working example.

[stefanprodan]

@relu if you can, please open a PR that moves the unit tests and git dirty check inside a container.

[phillebaba]

I can probably free up some time during the weekend but if @relu is already on it I can just rebase when his PR gets merged.

[phillebaba]

So I rebased the latest changes to run the test in a container. Could someone have a look and check if there is something else missing before this can be merged?

[phillebaba]

Just to make sure that we have discussed this point.

libgit2 will default to OpenSSL for HTTPS transport (except on Windows and macOS, as mentioned above). On any system, mbedTLS may be optionally enabled as the security provider. OpenSSL is thread-safe starting at version 1.1.0. If your copy of libgit2 is linked against that version, you do not need to take any further steps.
https://github.com/libgit2/libgit2/blob/master/docs/threading.md

Looking at alpine it should include openssl 1.1.1 so this should not be an issue.
https://pkgs.alpinelinux.org/packages?name=openssl&branch=v3.12

Could I just have someone else verify my thinking.

[relu]

I just checked the latest golang:1.15-alpine image, it looks like the package you're looking for is actually libssl1.1, which seems to be installed by default. Indeed its version matches 1.1.1 so it should be just fine according to the documentation.

# apk info libssl1.1
WARNING: Ignoring APKINDEX.2c4ac24e.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.40a3604f.tar.gz: No such file or directory
libssl1.1-1.1.1g-r0 description:
SSL shared libraries

libssl1.1-1.1.1g-r0 webpage:
https://www.openssl.org/

libssl1.1-1.1.1g-r0 installed size:
540672

[squaremo]

Just a couple of naming nits from me

[squaremo]

I reckon name the subpackages for the implementations.

[phillebaba]

Do you mean naming the package go-git instead of v1? Would that not create confusion as there would be two packages with the same name.

[hiddeco]

Why was this omitted?

[phillebaba]

Manifest gen requires libgit2 to be installed which wont work in the CI as this is run outside of the CI container. That is why it had to be removed.

[stefanprodan]

LGTM

Thanks @phillebaba 🥇

[relu]

LGTM 🦸

[dirien]

I really like this new property!

[dirien]

Question regarding the off the shelve (OTS) function from kustomize: Will this still work?

We use this quite heavily in our applications and found out in e.g. ArgoCD that even their switch to the native CLI it is still breaking in the off the shelve part of the kustomize.

Maybe it is more of a concern for kustomize cli.... ?