How do you manage zero downtime rollouts of this module?

[tmeijn]

Hey community! I was wondering how y'all are managing zero downtime when updating your GitLab Runner using Terraform? Do you have a nifty way to do zero downtime or do you just have scheduled maintenance windows? I'm curious what y'all managed to do regarding this subject. I myself have at one time done the following in a Terraform pipeline using Hetzner cloud:

1. Have one Terraform configuration with two partial backend states (main and transient). Transient had a different naming so I could distinguish the transient runner.
2. Spin up the "transient" infrastructure, monitor it was able to pick up jobs.
3. Pause the Main Runner and wait for main queue to finish using the GL api and curl/bash.
4. After main was idle, let pipeline run plan and apply.
5. Wait for transient queue to finish using same method.
6. Run terraform destroy on the transient infrastructure.

This worked pretty well, but doesn't feel like the best solution, so I'm curious what y'all have come up with.

[kayman-mk]

My simple solution is:

1. Wait until all developers are gone
2. Update the module from the local machine, e.g. terraform apply

Quite simple but not zero downtime as the Runners might be restarted due to changes.

For zero downtime you have to deploy the module twice during release time as you described above. One thing I am missing is, "setting the Runners to pause" in your description above.

[tmeijn]

One thing I am missing is, "setting the Runners to pause" in your description above.

Yeah that is done in step 3, I'll clarify.

Thanks for your reply. I made this pipeline for my personal GitLab group, but I notice now that it does not scale when having multiple Runner Managers in the same state. It'd be nice to have a way to do zero downtime, but this case simply patching after working hours seems way more simpler.

[kayman-mk]

Just found out, that idle on the Runners view in GitLab does not mean that no jobs are being processed by the Runner Workers. So you have to check every Runner Worker if there are still jobs being processed.

[kayman-mk]

Just dealing with this problem for our Live environment. The first concept looks like:

• have a dedicated project containing the critical build infrastructure, i.e. GitLab Runner, Docker, Maven, NPM repository manager.
• have a dedicated project containing one Runner only which is used to update the other project (let's call this project the "Updater")

The deployment of our critical infrastructure is disabled. We can merge into the main branch, but the changes are not delivered automatically. Early in the morning (let's say at 0400) we run a scheduled pipeline which is executed on the Updater. It deploys all changes which were merged the day before. This way our infrastructure is down at night/early morning only.

The Updater is a normal project, meaning that we merge and deploy whenever we like to (usually fully automatic). As it does not process any jobs during daytime, we do not care about a Runner restart. Our normal build infrastructure takes care of updating the Updater.

This way we can use our standard pipelines. There are only some minor modifications for the critical build infrastructure project (disable deployment, run a scheduled deployment at night, making sure that the deployment runs on the Updater). We will give it a try in the upcoming weeks.

[kayman-mk]

Since #1117 (May 2024) this has been solved. The old Runners are killed as soon as they finished the current jobs. The new Runners are spun up in parallel.