Improve k3s airgap boot time for large number/size of images

[harsimranmaan]

Is your feature request related to a problem? Please describe.

K3s loads images from the agent/images folder synchronously. As documented in rancher/rke2#773, it can be very slow to reboot/start k3s to get a functional control plane and data plane. The image load blocks until all img "sources" are loaded

Describe the solution you'd like

Provide a way to order the loading of system images(k3s, user provided) followed by other app images. Unblock k3s start and let all other images load in the background with a configurable number of goroutines. While this will not solve the problem when a giant image bundle is used but would help in cases where the bundle can be split and the chunks are loaded in a user-defined order with lazy-loading support. A simpler implementation that allows specifying the parallelization and an order based on file names would suffice to begin with.

Describe alternatives you've considered

Additional context

We are loading a 30GB zst compressed image bundle at k3s boot. The start takes >10mins. Splitting into multiple smaller bundles is not very helpful as the loading is not concurrent.

[brandond]

My recommendation for this in the past has been as follows:

1. Place only the REQUIRED images in the images dir
2. Manually load the remaining optional images via ctr using an ExecStartPost script referenced from the systemd unit.

The above is probably what I'll recommend when the distributed registry mirror from #8977 is available.

The best option when using a VERY large set of images is still to use a real registry.

[brandond]

I'm not convinced that K3s itself needs to have native support for staged and/or parallel import, given the options described above. Can you make a case for why this needs to be built in to k3s, given that the user has other options and can customize the import logic for their specific needs once containerd has been started?

[harsimranmaan]

My only arguments would be ease of use for the operator. Also, I am not sure that ctr supports importing zst compressed images. k3s provides an edge when it comes to have a simplified airgap experience. The only struggles have been around loading large bundles.

[brandond]

Something like this should work, assuming you have the zstd package for your distro installed.

ExecStartPost=for FILE in /var/lib/rancher/k3s/agent/extra-images/*.zst; do zstdcat $FILE | ctr -n k8s.io image import --all-platforms --no-unpack -; done

[brandond]

I think I will close this as unplanned at the moment, given the above workaround. but we can reevaluate in the future.

cc @caroline-suse-rancher