Support for containerd 1.5+ config_path: On-the-fly container registry configuration

[nickbp]

Is your feature request related to a problem? Please describe.

containerd versions 1.4 and older require adding any custom registry information via registry.mirrors/registry.configs sections in the main etc/containerd/config.toml. This mechanism has a few downsides, including that it requires editing the main containerd configuration in-place (which cannot be done in K3s since it'll be automatically re-rendered), and also it requires restarting containerd to take effect. This older mechanism is still available and is what k3s currently uses, but it has been deprecated.

With 1.5, containerd added support for a new config_path configuration option which imitates the behavior of dockerd's certs.d directory, with some additional functionality on top. This new system resolves the problems of the 1.4-and-earlier mechanism, making it very easy for an operator to manage custom registry configurations.

Describe the solution you'd like

K3s already exposes a custom registries.yaml configuration to add container registry configurations into containerd/config.toml. I think that the existing K3s configuration options could continue to be supported via the new config_path mechanism, where K3s would just write files into the config_path structure, rather than directly to containerd/config.toml. Meanwhile, users who wish to manage custom certs on-the-fly would be able to do so by adding them to the config_path directly on the host filesystem - e.g. via a DaemonSet with a hostPath mount, or externally via separate configuration management tooling (e.g. ansible or similar).

Implementation steps:

• The containerd/config.toml templates would be updated to specify a reasonable config_path path by default. For example, this could be etc/containerd/certs.d/ within the K3s install - so a new etc/containerd/certs.d directory next to the existing K3s-rendered etc/containerd/config.toml. It would probably make sense to allow customizing the config_path location via a flag and/or envvar.
• The existing containerd/config.toml PrivateRegistryConfig templating would be migrated to writing files into the configured config_path. This would effectively migrate users of registries.yaml to using config_path.

I think the K3s config_path output implementation should be permissive of any other "unknown" files that the user may have added to the directory. This has the potential downside of leaving "leftover" registry configurations lying around in config_path if they are removed from the user's registries.yaml configuration. The user would be able to delete these leftover files from config_path manually after removing them from registries.yaml.

Also, it may be worth thinking about deprecating the current registries.yaml mechanism entirely in favor of having users add their custom registry configs to config_path directly. Given that in-place editing of the main etc/containerd/config.toml would no longer be necessary under the new system, the custom registries.yaml mechanism wouldn't be providing much value. But there would be no rush since registries.yaml could continue to work on top of config_path as described above.

Describe alternatives you've considered

Alternative options:

• Using the current containerd/config.toml-based configuration mechanism exposed by K3s, requiring restarting the cluster after any changes.
• Configuring K3s to use dockerd and using its certs.d support.
• Providing a custom config.toml.tmpl which specifies config_path, effectively implementing the suggested config change manually.
• Disabling K3s-provided containerd in favor of a separate 1.5+ containerd with config_path enabled (but at this point why use K3s?)

Additional context

I encountered this when trying to set up a local Harbor registry. The most finicky part has been getting K3s' containerd-based kubelets to acknowledge the CA cert that was being used by Harbor. After looking through containerd docs, I noticed that they had recently added support for directory-based configuration which would allow me to automatically add the CA certs to the nodes without requiring reconfiguring and restarting the whole cluster.

Backporting

• Needs backporting to older releases

[brandond]

I will note that I'm not a huge fan of the new config_path mechanism - scattering files about the filesystem with names matching remote hosts doesn't feel very tidy to me, compared to having everything configured from a single config file. It is also more complicated to manage declaratively.

We can take a look at this, though.

[brandond]

From @ruifung in #7007:

Actual behavior:
It failed to pull images, and logged errors saying no authorization in containerd logs which suggests the rewrite was applied to the request to the original repository.

With the mirror configuration removed, it successfully pulls as expected.

What about supporting the hosts.toml mechanism and it's override-path option found in upstream containers?

Using hosts.toml would allow us to override paths. It's not as flexible as full rewriting, but it is per-endpoint, and probably accomplishes 99% of what people want out of rewrites.

[brandond]

@networkhermit your comments have nothing to do with this issue to track switching the config schema and structure. Fallback does work, with some restrictions on the rewrite behavior as described above. Please don't change the topic.

[networkhermit]

@brandond

Sorry about any misunderstanding. Maybe I should give more context.

As reported by #7007 and suggested in #7007 (comment), when using harbor registry as a registry mirror in k3s, the old rewrite mechanism would keep containerd from falling back to upstream endpoint when the mirror instance is down, and the workaround is manually adding config_path in /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl.

    81  {{ if .PrivateRegistryConfig }}
    82  [plugins."io.containerd.grpc.v1.cri".registry]
    83    config_path = "/etc/containerd/certs.d"
    84  {{ if .PrivateRegistryConfig.Mirrors }}

So I believe switching the config schema and structure is the final fix to that issue?

Am I supposed to comment in already closed #7007 in order to address my similar situation?

Edited:

As Harbor puts a registry name in the pull image path, one need to use the rewrite mechanism in k3s registries.yaml to make the registry mirror work.

[rlex]

What it will mean for end users? k3s will still be using single config as usual, but registries will be split into separate files in path specified in config_path? Old way of registries.yaml will be removed? Asking because i maintain k3s ansible role.

[brandond]

Take a look at the PR. It doesn't change anything regarding registries.yaml.

[aganesh-suse]

Validated on master branch with version v1.29.1+k3s2

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

system-default-registry: test-private-registry.com
debug: true
token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

Registries.yaml

$ cat /etc/rancher/rke2/registries.yaml
mirrors:
  test-private-registry.com:
    endpoint:
      - https://test-private-registry.com  
  docker.io:
    endpoint:
      - https://test-private-registry.com
  k8s.gcr.io:
    endpoint:
      - https://test-private-registry.com
configs:
  test-private-registry.com:
    auth:
      username: user
      password: password
    tls:
      ca_file: /home/ubuntu/ca.pem

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

Copy registries.yaml to /etc/rancher/k3s/registries.yaml
copy ca.pem to the user home directory (as per path provided in the registries.yaml file).
2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.29.1+k3s2' sh -s - server

3. Verify content of config.toml file for config_path:

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

4. Verify content of config_path directory and the corresponding host.toml files:

 $ sudo ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d
 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/test-private-registry.com/hosts.toml
 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.29.1+k3s2 (57482a1c)
go version go1.21.6
'''
Cluster Status:
'''
$ kubectl get nodes
NAME               STATUS   ROLES                       AGE     VERSION
ip-172-31-24-154   Ready    control-plane,etcd,master   4m19s   v1.29.1+k3s2
ip-172-31-24-217   Ready    control-plane,etcd,master   3m21s   v1.29.1+k3s2
ip-172-31-24-94    Ready    <none>                      2m41s   v1.29.1+k3s2
ip-172-31-29-236   Ready    control-plane,etcd,master   5m25s   v1.29.1+k3s2

$ kubectl get pods -A
NAMESPACE        NAME                                     READY   STATUS      RESTARTS   AGE
auto-clusterip   test-clusterip-7d68b9684-pl47v           1/1     Running     0          54s
auto-clusterip   test-clusterip-7d68b9684-rc9zp           1/1     Running     0          54s
auto-daemonset   test-daemonset-5mq7g                     1/1     Running     0          54s
auto-daemonset   test-daemonset-6zvt2                     1/1     Running     0          54s
auto-daemonset   test-daemonset-b922f                     1/1     Running     0          54s
auto-daemonset   test-daemonset-dkn42                     1/1     Running     0          54s
auto-dns         dnsutils                                 1/1     Running     0          54s
auto-ingress     test-ingress-9pdcp                       1/1     Running     0          53s
auto-ingress     test-ingress-bf95p                       1/1     Running     0          53s
auto-nodeport    test-nodeport-66855bd4b4-78fq9           1/1     Running     0          53s
auto-nodeport    test-nodeport-66855bd4b4-hcqmj           1/1     Running     0          53s
default          clusterip-pod-demo                       1/1     Running     0          57s
default          clusterip-pod-demo-2                     1/1     Running     0          57s
default          clusterip-pod-demo-3                     1/1     Running     0          57s
kube-system      coredns-b85bd4cbf-7vf5g                  1/1     Running     0          4m47s
kube-system      helm-install-traefik-crd-zbxn9           0/1     Completed   0          4m48s
kube-system      helm-install-traefik-q77d5               0/1     Completed   1          4m48s
kube-system      local-path-provisioner-d4fc45d4c-98bz9   1/1     Running     0          4m47s
kube-system      metrics-server-6bb469c79b-mqzcd          1/1     Running     0          4m47s
kube-system      svclb-traefik-c5ea0062-cfz9v             2/2     Running     0          4m18s
kube-system      svclb-traefik-c5ea0062-pc64k             2/2     Running     0          2m40s
kube-system      svclb-traefik-c5ea0062-qrbmn             2/2     Running     0          3m20s
kube-system      svclb-traefik-c5ea0062-sm4cg             2/2     Running     0          4m33s
kube-system      traefik-66ff7c6c94-fgbrs                 1/1     Running     0          4m34s
pvt-reg-test     pvt-reg-test-6ccdf98b56-f6cf5            1/1     Running     0          40s
pvt-reg-test     pvt-reg-test-6ccdf98b56-hmj2g            1/1     Running     0          40s

config.toml content:

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

.
.
.

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"




[plugins."io.containerd.grpc.v1.cri".registry.configs."test-private-registry.com".auth]
  username = "user"
  password = "password"

certs.d directory content:

 $ sudo ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d
docker.io
test-private-registry.com
k8s.gcr.io

hosts.toml file contents:

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/test-private-registry.com/hosts.toml
# File generated by k3s. DO NOT EDIT.


[host."https://test-private-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"

[host."https://test-private-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.
server = "https://k8s.gcr.io/v2"

[host."https://test-private-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]