Matrix k8s version testing [latest, n-1] on k3d

[MxNxPx]

Describe what should be investigated or refactored

Currently we test on a single k3d version in PRs. In order to expand test coverage to handle other k8s versions we should matrix this test across several minor versions. To start this should be for k3d only, and cover the latest minor version + n-1 minor version. To reduce PR CI time this should only be done nightly to start.

Definition of done: Addition of nightly testing on matrixed k3d versions, covering n-1 and latest. Ideally this is easily extensible to additional versions in the future (matrixed, task inputs, etc).

Additional context

Considering there will be folks using UDS on older K8s versions than what is used to publish, ought to consider adding some testing of n-1 K8s version.

[mjnagel]

Updating this ticket:

• we are 1.29.5 for k3d/k3s
• we are on 1.27 for EKS
• RKE2 infra is not maintained as a part of our repo/CI currently but we have other testing environments that we should update to 1.29+

[mjnagel]

Rescoping this effort to focus on testing across multiple k8s distros. To start we should do nightly test on the latest k8s minor version and n-1 minor version, at minimum on k3d - possibly on eks as well.

[MxNxPx]

Updating this ticket:

• we are 1.29.5 for k3d/k3s
• we are on 1.27 for EKS
• RKE2 infra is not maintained as a part of our repo/CI currently but we have other testing environments that we should update to 1.29+

Now on 1.30 for EKS per #605

[mjnagel]

Updated ticket description to focus on k3d, nightly, and 2 versions to start.

[noahpb]

I would add that our nightly ci tests across the various k8s distros (k3d, rke2, eks, aks) should all run against the same minor version of k8s. I added an issue to track this here #784.