New image id with every build

[dciancu]

Description

FROM debian
RUN echo 1

docker build -t test . results in an image id
running docker build -t test . again gives a new image id (despite saying CACHED 2/2)

Reproduce

Use any simple Dockerfile and run docker build twice for it, you will get two different image ids

Expected behavior

docker build should not generate a new image id if all steps are cached

docker version

Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.22.3
 Git commit:        b72abbb6f0
 Built:             Thu May 16 07:47:24 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:04 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.1.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     /Users/dciancu/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     /Users/dciancu/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.29
    Path:     /Users/dciancu/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/dciancu/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/dciancu/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/dciancu/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/dciancu/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/dciancu/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     /Users/dciancu/.docker/cli-plugins/docker-scout

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 26.1.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.26-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 3.831GiB
 Name: docker-desktop
 ID: cb1ff28e-f919-4278-bb9f-10dcfe111250
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/dciancu/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

080414E0-4A72-428B-B7E8-64293C8ED6A1/20240527075817

Additional Info

No response

[thaJeztah]

I tried reproducing, but didn't manage to.

mkdir repro-7297 && cd repro-7297
echo -e "FROM debian\nRUN echo 1\n" > Dockerfile

docker build -t test .
...
 => [2/2] RUN echo 1                                                                             0.3s
 => exporting to image                                                                           0.0s
 => => exporting layers                                                                          0.0s
 => => writing image sha256:3a097764d6b73e779c6d27dc92a09837bcc950def76056e8451d86d104297b7b     0.0s
 => => naming to docker.io/library/test                                                          0.0s

docker image inspect --format '{{.ID}}' test
sha256:3a097764d6b73e779c6d27dc92a09837bcc950def76056e8451d86d104297b7b

docker build -t test .
...
 => CACHED [2/2] RUN echo 1                                                                       0.0s
 => exporting to image                                                                            0.0s
 => => exporting layers                                                                           0.0s
 => => writing image sha256:3a097764d6b73e779c6d27dc92a09837bcc950def76056e8451d86d104297b7b      0.0s
 => => naming to docker.io/library/test                                                           0.0s

docker image inspect --format '{{.ID}}' test
sha256:3a097764d6b73e779c6d27dc92a09837bcc950def76056e8451d86d104297b7b

docker build -t test .
...
 => CACHED [2/2] RUN echo 1                                                                       0.0s
 => exporting to image                                                                            0.0s
 => => exporting layers                                                                           0.0s
 => => writing image sha256:3a097764d6b73e779c6d27dc92a09837bcc950def76056e8451d86d104297b7b      0.0s
 => => naming to docker.io/library/test                                                           0.0s

docker image inspect --format '{{.ID}}' test
sha256:3a097764d6b73e779c6d27dc92a09837bcc950def76056e8451d86d104297b7b

Are you using the default builder, or is it possible you created a custom (container-driver) builder? What does docker buildx inspect show?

[thaJeztah]

cc @crazy-max

[dciancu]

I should add this is happening on Apple M2 chip.
I can reproduce with or without Rosetta emulation.

docker buildx inspect

Name:          desktop-linux
Driver:        docker
Last Activity: 2024-05-28 13:05:46 +0000 UTC

Nodes:
Name:             desktop-linux
Endpoint:         desktop-linux
Status:           running
BuildKit version: v0.13.2
Platforms:        linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
Labels:
 org.mobyproject.buildkit.worker.containerd.namespace: moby
 org.mobyproject.buildkit.worker.containerd.uuid:      7c6741d7-bcb9-4fae-a25c-c255c1d97b32
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.hostname:             docker-desktop
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 192.168.65.254
 org.mobyproject.buildkit.worker.network:              host
 org.mobyproject.buildkit.worker.selinux.enabled:      false
 org.mobyproject.buildkit.worker.snapshotter:          overlayfs
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    2.764GiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    20GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 20GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 20GiB

$ docker build -t test .
[+] Building 1.3s (7/7) FINISHED                           docker:desktop-linux
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 137B                                       0.0s
 => [internal] load metadata for docker.io/library/debian:latest           1.2s
 => [auth] library/debian:pull token for registry-1.docker.io              0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [1/2] FROM docker.io/library/debian:latest@sha256:fac2c0fd33e88dfd3bc  0.0s
 => => resolve docker.io/library/debian:latest@sha256:fac2c0fd33e88dfd3bc  0.0s
 => CACHED [2/2] RUN echo 1                                                0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => exporting manifest sha256:eb9b31e6cc6ba18774f2bac02b1a2830143e3d17  0.0s
 => => exporting config sha256:465c7fde00b1efc655f1dfa9d66b3436f066f91eb5  0.0s
 => => exporting attestation manifest sha256:1d3478ffdd28421b3fa7336e6a6d  0.0s
 => => exporting manifest list sha256:c81d9b0d12d2f7de34f04cdbf55a4270b4b  0.0s
 => => naming to docker.io/library/test:latest                             0.0s
 => => unpacking to docker.io/library/test:latest

$ docker image inspect --format '{{.ID}}' test
sha256:c81d9b0d12d2f7de34f04cdbf55a4270b4bb686e9c7f6a8ebbfc861b8702dbe0

$ docker build -t test .
[+] Building 0.6s (6/6) FINISHED                           docker:desktop-linux
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 137B                                       0.0s
 => [internal] load metadata for docker.io/library/debian:latest           0.5s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [1/2] FROM docker.io/library/debian:latest@sha256:fac2c0fd33e88dfd3bc  0.0s
 => => resolve docker.io/library/debian:latest@sha256:fac2c0fd33e88dfd3bc  0.0s
 => CACHED [2/2] RUN echo 1                                                0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => exporting manifest sha256:eb9b31e6cc6ba18774f2bac02b1a2830143e3d17  0.0s
 => => exporting config sha256:465c7fde00b1efc655f1dfa9d66b3436f066f91eb5  0.0s
 => => exporting attestation manifest sha256:4da82f4c2e79738e306d2002e5f4  0.0s
 => => exporting manifest list sha256:5ad2d08521aaa4c7be79cfc5b2060b21056  0.0s
 => => naming to docker.io/library/test:latest                             0.0s
 => => unpacking to docker.io/library/test:latest

$ docker image inspect --format '{{.ID}}' test
sha256:5ad2d08521aaa4c7be79cfc5b2060b21056d6b64488beeb95ab0c11eb8cb55c7

$ docker build -t test .
[+] Building 0.6s (6/6) FINISHED                           docker:desktop-linux
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 137B                                       0.0s
 => [internal] load metadata for docker.io/library/debian:latest           0.5s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [1/2] FROM docker.io/library/debian:latest@sha256:fac2c0fd33e88dfd3bc  0.0s
 => => resolve docker.io/library/debian:latest@sha256:fac2c0fd33e88dfd3bc  0.0s
 => CACHED [2/2] RUN echo 1                                                0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => exporting manifest sha256:eb9b31e6cc6ba18774f2bac02b1a2830143e3d17  0.0s
 => => exporting config sha256:465c7fde00b1efc655f1dfa9d66b3436f066f91eb5  0.0s
 => => exporting attestation manifest sha256:3faa247b08b5e1828266ed1d6497  0.0s
 => => exporting manifest list sha256:0079f4ba3f50a0fd2110d36e5dfdfb1f01a  0.0s
 => => naming to docker.io/library/test:latest                             0.0s
 => => unpacking to docker.io/library/test:latest

$ docker image inspect --format '{{.ID}}' test
sha256:0079f4ba3f50a0fd2110d36e5dfdfb1f01ac6edba3e4638164f3ae1f7f831fbb

From above output, attestation manifest seems to be changing.

[thaJeztah]

Ah, thanks for that output! Yes, it looks like the image doesn't change, but the attestation gets created for each build, so the digest of the manifest index will change because of that;

Image digest is the same for all builds;

 => => exporting manifest sha256:eb9b31e6cc6ba18774f2bac02b1a2830143e3d17  0.0s

cc @tonistiigi @crazy-max PTAL

[dciancu]

Done some further debugging and it is caused by containerd image store.
When this is enabled, the attestation gets created for each build, no matter what builder is being used.
When disabled, attestation manifest seems to not get created, and thus the manifest index will not change.
Anyhow, you got it right.

[crazy-max]

When using the containerd image store an OCI index is pushed that contains the provenance attestation as this store supports multi-platform images. This is the case since Buildx 0.10.0:

Buildx v0.10 enables support for a minimal SLSA Provenance attestation, which requires support for OCI-compliant multi-platform images. This may introduce issues with registry and runtime support (e.g. docker/buildx#1533). You can optionally disable the default provenance attestation functionality using --provenance=false.

I recall someone was also confused about the image id being different with provenance because of lack of reproducibility/immutability of it but can't find yet the issue related to this. In any case you can just disable provenance if this is an issue for you:

docker build -t test --provenance=false .

[dciancu]

OK understood, thank you very much for clarifying this.
Yes, indeed, it is confusing.
I think what is changing in the attestation are the timestamps, maybe they can be reused (cached) if all other values are the same and all the image steps are cached?