Mirror image rewrites break fallback to original repository

[ruifung]

Environmental Info:
K3s Version: v1.25.6+k3s1 (9176e03)

Node(s) CPU architecture, OS, and Version: Linux server-pi-01 5.15.0-1024-raspi #26-Ubuntu SMP PREEMPT Wed Jan 18 15:29:53 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: 1 server, single node.

Describe the bug:

When attempting to use registry mirrors with rewrites (to a local harbor installation hosted on the node itself) fallback to upstream repositories will fail with errors logged in the containerd log file that suggests the rewrites are being applied to queries sent to the original registry.

This prevents it from gracefully falling back on the original registry when using mirrors as a caching source.

Steps To Reproduce:

• Installed K3s:
• Disabled traefik and servicelb via systemd overrides
• Added registries.yaml with content
• Try to pull an image

mirrors:
  docker.io:
    endpoint:
      - https://harbor.services.home.arpa
    rewrite:
        "^(.*)$": "dockerhub/$1"

Expected behavior:

For it to successfully pull images from the original repository when the registry is offline.

Actual behavior:

It failed to pull images, and logged errors saying no authorization in containerd logs which suggests the rewrite was applied to the request to the original repository.

With the mirror configuration removed, it successfully pulls as expected.

Additional context / logs:

time="2023-03-01T16:27:18.776856761+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://harbor.services.home.arpa/v2/dockerhub/library/busybox/manifests/latest?ns=docker.io\": dial tcp 10.229.0.5:443: connect: no route to host" host=harbor.services.home.arpa
time="2023-03-01T16:27:20.787335780+08:00" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
time="2023-03-01T16:27:20.802273766+08:00" level=error msg="PullImage \"busybox:latest\" failed" error="failed to pull and unpack image \"docker.io/library/busybox:latest\": failed to resolve reference \"docker.io/library/busybox:latest\": failed to do request: Head \"https://harbor.services.home.arpa/v2/dockerhub/library/busybox/manifests/latest?ns=docker.io\": dial tcp 10.229.0.5:443: connect: no route to host"

[brandond]

Yes, this is an issue with rewrites. They probably should have been made per endpoint rather than per registry. They also probably should have been an ordered list instead of a map[string]string. Fixing either of these problems in a backwards compatible way will need to be managed carefully though.

[ruifung]

What about supporting the hosts.toml mechanism and it's override-path option found in upstream containers?

That works as expected with harbor as a proxy and will correctly fall back to the upstream source, judging by my experience with that on another cluster running Talos Linux.

[ruifung]

For anyone who stumbles upon this:

I've worked around it by simply not using k3s registries and rewrites configuration and modifying the containerd config instead to use the hosts.toml mechanism instead. Documentation can be found here:

• https://docs.k3s.io/advanced#configuring-containerd
• https://github.com/containerd/containerd/blob/main/docs/hosts.md

[brandond]

There's an existing issue for switching registries.yaml to use hosts.toml instead, I'll probably close this in favor of tracking on that one - but I'll get it linked up first.

[brandond]

Will track in #5568