not found error while creating manifest list for docker.io/rancher/mirrored-pause:3.8

[superseb]

Seen in https://drone-publish.rancher.io/rancher/image-mirror/754/1/3:

Writing manifest to image destination
Storing signatures
	Unchanged: registry.k8s.io/pause@sha256:566af08540f378a70a03588f3963b035f33c49ebab3e4e13a4f5edbcd78c6689 == docker.io/rancher/mirrored-pause:3.8-arm64
	           sha256:566af08540f378a70a03588f3963b035f33c49ebab3e4e13a4f5edbcd78c6689
	Unchanged: registry.k8s.io/pause@sha256:27295ffe5a75328e8230ff9bcabe2b54ebb9079ff70344d73a7b7c7e163ee1a6 == docker.io/rancher/mirrored-pause:3.8-arm
	           sha256:27295ffe5a75328e8230ff9bcabe2b54ebb9079ff70344d73a7b7c7e163ee1a6
	Unchanged: registry.k8s.io/pause@sha256:7eaeb31509d7f370599ef78d55956e170eafb7f4a75b8dc14b5c06071d13aae0 == docker.io/rancher/mirrored-pause:3.8-s390x
	           sha256:7eaeb31509d7f370599ef78d55956e170eafb7f4a75b8dc14b5c06071d13aae0
	Writing manifest list to docker.io/rancher/mirrored-pause:3.8
sha256:27295ffe5a75328e8230ff9bcabe2b54ebb9079ff70344d73a7b7c7e163ee1a6
sha256:566af08540f378a70a03588f3963b035f33c49ebab3e4e13a4f5edbcd78c6689
sha256:78bfb9d8999c190fca79871c4b2f8d69d94a0605266f0bbb2dbaa1b6dfd03720
sha256:7eaeb31509d7f370599ef78d55956e170eafb7f4a75b8dc14b5c06071d13aae0
sha256:9d05676469a08d6dba9889297333b7d1768e44e38075ab5350a4f8edd97f5be1
sha256:e8fb66bcfe1a85ec1299652d28e6f7f9cfbb01d33c6260582a42971d30dcb77d
sha256:f5944f2d1daf66463768a1503d0c8c5e8dde7c1674d3f85abc70cef9c7e32e95
httpReaderSeeker: failed open: content at https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/sha256:78bfb9d8999c190fca79871c4b2f8d69d94a0605266f0bbb2dbaa1b6dfd03720 not found: not found
===
Failed copying image for rancher/mirrored-pause
===

I'm not sure if this is intentional, but with multiple amd64 images (linux and windows), the logic is kinda looping through all found digests and it looks up the current digest to diff with the found digest, and as it differs, it syncs all the digests needed for amd64. It doesn't look like it was meant to be that way for windows, but it works. Except for this case, as the resulting digest after running skopeo copy is different. This is what I saw:

bash-5.0# skopeo inspect docker://registry.k8s.io/pause@sha256:85cfebc79dccc7a0e56680778a614b29a0b1c2ae98d4b1efc746764c04d5656c --raw 2>/dev/null | sha256sum 
85cfebc79dccc7a0e56680778a614b29a0b1c2ae98d4b1efc746764c04d5656c  -
bash-5.0# skopeo inspect docker://docker.io/superseb/mirrored-pause:3.7-amd64  --raw 2>/dev/null | sha256sum
85cfebc79dccc7a0e56680778a614b29a0b1c2ae98d4b1efc746764c04d5656c  -
bash-5.0# skopeo inspect docker://registry.k8s.io/pause@sha256:e8fb66bcfe1a85ec1299652d28e6f7f9cfbb01d33c6260582a42971d30dcb77d --raw 2>/dev/null | sha256sum
e8fb66bcfe1a85ec1299652d28e6f7f9cfbb01d33c6260582a42971d30dcb77d  -
bash-5.0# skopeo inspect docker://docker.io/superseb/mirrored-pause:3.8-amd64  --raw 2>/dev/null | sha256sum
01c7ed0d93c697230ca7b0373f9ae34e25fc8cbbea084191c59fe1108f0e31a1  -

And the diff was showing two differences:

1. The response was newlined (origin) and not newlined when querying DockerHub (single line JSON)
2. Only a few layers are gzipped (origin) vs all being gzipped on DockerHub

skopeo inspect docker://registry.k8s.io/pause@sha256:e8fb66bcfe1a85ec1299652d28e6f7f9cfbb01d33c6260582a42971d30dcb77d --raw 
{
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "schemaVersion": 2,
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "digest": "sha256:7effaf5879989caec6af88410f43cb0aac21f7c0b65a966a1977a1e14f6ecf2e",
      "size": 2610
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
         "digest": "sha256:26085ddda8400952278f21eda4de3abb08e6e2b02afaddd77d1dab2b2a3f7c70",
         "size": 303383040
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
         "digest": "sha256:61b9d3e54fdf80b25126caf3898b7619f2cc4271b3707e147145c6aa7d7b6663",
         "size": 56320
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
         "digest": "sha256:2a2a37705371efd292777b63ae5500db8c667ea6a2269db5a82bd7ddb37c611d",
         "size": 52736
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
         "digest": "sha256:51433a8bc864e113031f56050243d96f07ee45ee87cfb769f94878f710fcfc00",
         "size": 1379328
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
         "digest": "sha256:ac0271ec295db8efe49fd6d64caa24b7612828d8de3596df8118085a7f9c9f77",
         "size": 1286656
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
         "digest": "sha256:0a1660a7fc0943f082ad82712f70f1f9864c10fcd7b6c2aa34b66e320d28df50",
         "size": 52736
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:c44b3b94f1619a178691090b2a02c123a5354f863497e64e3cc7b9f77c55ba79",
         "size": 20774
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "digest": "sha256:77fb4c7d81f02ad14b711a776993cd511e78545c5e6416dd91de07d8ebf9a787",
         "size": 1402287
      }
   ]
}

skopeo inspect docker://docker.io/superseb/mirrored-pause:3.8-amd64  --raw 2>/dev/null       
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":2610,"digest":"sha256:7effaf5879989caec6af88410f43cb0aac21f7c0b65a966a1977a1e14f6ecf2e"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":125575393,"digest":"sha256:74930762f9101c7fdb6be93708b93e0bd982831c7feecfc60863d126066dd0df"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1890,"digest":"sha256:553dc5cabf3637632c6d5ea0ed3a417b301c641191823d86ef3058d6279c5492"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1353,"digest":"sha256:966a9745c8adacd90195a617131d8ee7f1037ee2574c21ca70605c658195b59d"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":90718,"digest":"sha256:6e157444b00193a8f77eaba8293b00b9c45faf2df32de43cec53e6cf3aaf97c8"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":56273,"digest":"sha256:d5434e2517d7db7fede0fd93607d62d305f29d3ad0ca78ee85407f8390e2a612"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1357,"digest":"sha256:1d8c49f4e7a53129f5e03d26ae07373eb74ff5b49ef1255495fd3930f5e4cc4b"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":20774,"digest":"sha256:c44b3b94f1619a178691090b2a02c123a5354f863497e64e3cc7b9f77c55ba79"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1402287,"digest":"sha256:77fb4c7d81f02ad14b711a776993cd511e78545c5e6416dd91de07d8ebf9a787"}]}

There are a few things we need to know before we can proceed to a solution:

• Is/was Windows part of the initial implementation and does it account for multiple -amd64 images (one for Linux and multiple for Windows), this is probably best answered by @brandond

Possible solutions:

• I tested upgrading skopeo and using --preserve-digests, this will solve the problem for this image but needs testing against the whole list
• We can move to --all when the source image/manifests is a list and just sync the whole list at once. This was tested for pause:3.8 image but also needs testing for others. This needs input from @brandond as well if this was considered before (one "downside" I guess is that it syncs all archs and not just the ones we choose)

[brandond]

• Is/was Windows part of the initial implementation and does it account for multiple -amd64 images (one for Linux and multiple for Windows)
  No, it wasn't in scope and it probably isn't handled properly.
• We can move to --all when the source image/manifests is a list and just sync the whole list at once
  I am not sure if this was documented anywhere, but I was asked to only mirror supported architectures. I don't see any reason why it would cause problems, other than people getting nervous when all the old tags change due to the manifest lists being updated to add architectures that were previously skipped.

[superseb]

So the easiest and quickest solution would be to move to skopeo copy --all --preserve-digests whenever the mediaType is "application/vnd.docker.distribution.manifest.list.v2+json" or "application/vnd.oci.image.index.v1+json". Although the digest changes, the layers remain the same so nodes pulling will not pull anything new but the digest will change. This is technically not a problem but we do publish digests for each release...

The safest option would be to apply the new flags only to new images, which would require something to differentiate the new images from the old. I guess not to freak out users, we can copy the current list and compare the images with the copied list to see if they are new and apply the new mechanism, else continue to use the current mechanism. This way the published digests stay relevant, and we can use the new mechanism on new images.

I guess this needs some more eyes/thoughts, if you have time @kinarashah @brandond.

[kinarashah]

@superseb I agree that the idea to apply new flags only to new images would be safer. Not sure if we want to use --all and push images for all archs under rancher/ when we don't technically support them.

As of now the issue was only with pause image but since the fix is generic, @cbron could you also take a look at this issue?

[superseb]

Any objection with the approach as stated in #379 (comment) ? @brandond @kinarashah

Using --all will simplify the script and speed up the process, downside is that all archs will be mirrored instead of what we specify. Just need to figure out how we want to split old vs new approach, copy images-list to a different filename and apply the current logic to images that exist in the different filename and new logic to those that not exist in the different filename?