[EPIC] Move build/release automation to GH Actions

[galal-hussein]

This epic issue is to track the effort of moving the automation of build and release of the following repositories to Github action instead of Drone:

• Use GH actions for build/release - image-build-base #379
• Use GH actions for build/release - image-build-kubernetes #394
• Use GH actions for build/release - image-build-rke2-cloud-provider #398
• Use GH actions for build/release - image-build-flannel #380
• Use GH actions for build/release - image-build-calico #381
• Use GH actions for build/release - image-build-coredns #382
• Use GH actions for build/release - image-build-multus #384
• Use GH actions for build/release - image-build-whereabouts #390
• Use GH actions for build/release - image-build-k8s-metrics-server #383
• Use GH actions for build/release - image-build-cluster-proportional-autoscaler #391
• Use GH actions for build/release - image-build-dns-nodecache #392
• Use GH actions for build/release - image-build-containerd  #399
• Use GH actions for build/release - image-build-runc #400
• Use GH actions for build/release - SRIOV images #401
• Use GH actions for build/release - rke2 #402
• Use GH actions for build/release - wharfie #403
• Use GH actions for build/release - rke2-upgrade #404
• Use GH actions for build/release - rke2-selinux #405
• Use GH actions for build/release - rke2-packaging #406
• Use GH actions for build/release - migration-agent #407
• Use GH actions for build/release - k3k #408
• Use GH actions for build/release - image-build-skel #409
• Use GH actions for build/release - image-build-node-feature-discovery #410
• Use GH actions for build/release - image-build-kube-proxy #411
• Use GH actions for build/release - image-build-helm-controller #412
• Use GH actions for build/release - image-build-etcd #413
• Use GH actions for build/release - image-build-crictl #414
• Use GH actions for build/release - image-build-coreos-flannel #415
• Use GH actions for build/release - image-build-cni-plugins #416
• Use GH actions for build/release - image-build-cilium #417
• Use GH actions for build/release - image-build-cilium-image-tools #418
• Use GH actions for build/release - ingress-nginx #427
• Use GH actions for build/release - rancher/rancher #425

The migration should involve migrating the PR drone and push drone as well, which will include:

• indicating successful/failure builds
• in case of release:
  • release any related artifacts to the github release
  • the push of the related docker images to docker hub

Decisions taken so far (up for discussion, of course!):

• Use buildx to build ARM64. General guidance from RAT suggests to wait until we have our own ARM64 runners because or limited resources but RAT is ok for our projects to use buildx as the resource consumption is not high
• Separate the build in two different jobs: one for amd64 and one for arm64. The reason is that we are running trivy to inspect the images and thus they need to be locally available. Combining both archs in one job is not supported because buildx and options like --load or outputs: type=docker are not compatible with multi-arch images
• In the push job, we can combine both archs as we are not applying trivy
• Keep the Makefile we were using with drone, even if we are not using it anymore. This can help run the build commands manually
• In the build workflow, consume the TAG from the Makefile. Creating the tag: version-build$DATE is complicated in github actions and we already have that in the Makefile
• Security recommends only running trivy in one arch, it does not have to be in both (image-build-flannel is using it in both)
• Security recommends using the trivy github action aquasecurity/trivy-action + dependabot as opposed to running the trivy binary inside the build-base image. This way it is easy to make sure that we are on the latest trivy code
• Security recommends adding to the workflows:

permissions:
    contents: read
    security-events: write # upload Sarif results (only in build workflow)

• Security recommends uploading the detected CVEs with github/codeql-action/upload-sarif@v3 to keep track of them

We need EIO:

• Add secrets to Github Actions: https://github.com/rancherlabs/eio/issues/2106, https://github.com/rancherlabs/eio/issues/2127
• Remove drone webhooks: (no issue yet)

GOTCHAS:

• Remember to remove the updatecli policy which updates drone

[manuelbuil]

There are a few more projects:

• https://github.com/rancher/image-build-k8s-metrics-server
• https://github.com/rancher/image-build-cluster-proportional-autoscaler
• https://github.com/rancher/image-build-dns-nodecache
• all sriov related projects
• https://github.com/rancher/image-build-containerd ?
• https://github.com/rancher/image-build-runc ?

[briandowns]

https://github.com/rancher/ecm-distro-tools/blob/master/docs/rke2-dependencies.md

[manuelbuil]

https://github.com/rancherlabs/eio/issues/2106

[galal-hussein]

I will add the rest of the images to the list, so that we can create related issues for them, also we would need to document the process here so that we have consistency between the repos

[cwayne18]

Are we also doing the switch to either scratch or bci-busybox at teh same time?

[galal-hussein]

Are we also doing the switch to either scratch or bci-busybox at the same time?

I don't think we should, I think these series of PRs can act as a template for any other project that we need to move to GH action, so I think its better if we separete the moving to bci-busybox to another series of PRs

[caroline-suse-rancher]

The idea right now, is that this effort will be complete by June sometime, though I know we're blocked on an EIO step right now. I will update here as I know more. Thank you all!

[galal-hussein]

This is a small guide that should be applied for any new image/repo that we want to migrate to GH Actions:

Image repos (image-build-*)

---

Step 1 - Remove Drone Traces

All traces of drone must be removed, that includes:

• .drone.yml file
• TAG ?= ${DRONE_TAG} in the makefile
• updatecli - remove drone if mentioned from updatecli

Step 2 - Remove manifest

Since we are using buildx now to support multiarch builds, then there is no need for manifest.tmpl, this should be removed from:

• Makefile:

.PHONY: image-manifest
image-manifest:
	DOCKER_CLI_EXPERIMENTAL=enabled docker manifest create --amend \
		$(ORG)/hardened-<image>:$(TAG) \
		$(ORG)/hardened-<image>:$(TAG)-$(ARCH)
	DOCKER_CLI_EXPERIMENTAL=enabled docker manifest push \
		$(ORG)/hardened-<image>:$(TAG)

• manifest.tmpl file

Step 3 - Add log to the makefile

The idea of the migration is simple, basically includes adding a log section to the makefile which will print all important environment variables that are given by tags or PRs and will be used by the github actions:
Makefile

.PHONY: log
log:
	@echo "ARCH=$(ARCH)"
	@echo "TAG=$(TAG)"
	@echo "ORG=$(ORG)"
	@echo "PKG=$(PKG)"
	@echo "SRC=$(SRC)"
	@echo "BUILD_META=$(BUILD_META)"
	@echo "K3S_ROOT_VERSION=$(K3S_ROOT_VERSION)"
	@echo "UNAME_M=$(UNAME_M)"

And then it can be used in the github action as follows:

run: |
        echo "$(make -s log | grep TAG)" >> "$GITHUB_ENV"

Step 4 - Add .github/workflow files

For image repos we should have 2 main files:

1- .github/workflows/build.yml (see flannel repo as a reference)
2- .github/workflows/image-push.yml (see flannel repo as a reference)

The build.yml file simply builds the images for amd64, arm64 architectures and then runs trivy for scans, and image-push does the same thing but with push set to true.

Note that you would need to set the proper permissions on the pipeline for the image-push workflow, this happens through pulling the secret from vault as follows:

secrets: |
          secret/data/github/repo/${{ github.repository }}/dockerhub/${{ github.repository_owner }}/credentials username | DOCKER_USERNAME ;
          secret/data/github/repo/${{ github.repository }}/dockerhub/${{ github.repository_owner }}/credentials password | DOCKER_PASSWORD

[thomasferrandiz]

@galal-hussein very good summary, that's exactly what we did in the PRs already merged

[manuelbuil]

great summary! Maybe warn users that the way to fetch dockerhub secrets changed. Now we use a EIO github action and read the username and password from an env variable