Image built by kaniko might have whiteout files that *seems* unnecessary

[ruiwen-zhao]

Actual behavior
Image built by kaniko might have whiteout files that seems unnecessary. More details in the To Reproduce section below. This behavior caused image pulling error when the image is pulled and unpacked by containerd. See containerd/containerd#4659

Expected behavior
Image built by kaniko should not have whiteout files if the actual files or the directories are deleted on lower layers.

To Reproduce
See containerd/containerd#4659 (comment) for details , but in general:

1. Build an image with the following Dockerfile

FROM fluent/fluentd:v1.9.3-1.0

USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install \
        fluent-plugin-nsq:0.0.6 \
        fluent-plugin-kubernetes_metadata_filter:2.4.1 \
        fluent-plugin-record-modifier:2.1.0 \
        fluent-plugin-prometheus:1.7.3 \
        fluent-plugin-throttle:0.0.4 \
        fluent-plugin-multi-format-parser:1.0.0 \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

RUN gem install nsq-ruby:2.3.1 && gem uninstall -I nsq-ruby --version 1.7.0

2. Build the image with kaniko

tar -cf - Dockerfile | gzip -9 | docker run   --interactive -v `pwd`/config.json:/kaniko/.docker/config.json:ro -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest   --context tar://stdin --destination qiutongs/kaniko-example:latest --force

3. Observe that top layer of the kaniko-built image is different than the image with same Dockerfile built by docker

Kaniko built:

root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-1.24.2.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-http-0.50.0.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-io-1.27.3.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-pool-0.2.0.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.bigdecimal-1.4.4.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.concurrent-ruby-1.1.6.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.console-1.8.2.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.cool.io-1.6.0.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.ext_monitor-0.1.2.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.fluentd-1.9.3.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.http_parser.rb-0.6.0.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.json-2.3.0.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.msgpack-1.3.3.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.nio4r-2.5.2.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.oj-3.8.1.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-hpack-1.4.2.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-http-0.13.1.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-http1-0.10.2.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-http2-0.10.4.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.serverengine-2.2.1.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.sigdump-0.2.4.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.strptime-0.2.3.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.timers-4.3.0.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.tzinfo-2.0.1.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.tzinfo-data-1.2019.3.gemspec
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.yajl-ruby-1.4.1.gemspec
usr/lib/ruby/gems/2.5.0/gems/.wh.nsq-ruby-1.7.0
usr/lib/ruby/gems/2.5.0/specifications/.wh.nsq-ruby-1.7.0.gemspec
/
root/
root/.gem/
root/.gem/specs/
root/.gem/specs/api.rubygems.org%443/
root/.gem/specs/api.rubygems.org%443/quick/
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/nsq-ruby-2.3.1.gemspec
...

Docker built:

root/
root/.gem/
root/.gem/specs/
root/.gem/specs/api.rubygems.org%443/
root/.gem/specs/api.rubygems.org%443/quick/
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/
root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/nsq-ruby-2.3.1.gemspec
...

Note the whiteout files (e.g. root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-1.24.2.gemspec) on the top layer of the kaniko-built image. The directory has actually been deleted in lower layers, so there is no need to have these whiteout files on the top layer. (And that's why the docker-built image doesn't have them)

Additional Information

• Dockerfile

FROM fluent/fluentd:v1.9.3-1.0

USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install \
        fluent-plugin-nsq:0.0.6 \
        fluent-plugin-kubernetes_metadata_filter:2.4.1 \
        fluent-plugin-record-modifier:2.1.0 \
        fluent-plugin-prometheus:1.7.3 \
        fluent-plugin-throttle:0.0.4 \
        fluent-plugin-multi-format-parser:1.0.0 \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

RUN gem install nsq-ruby:2.3.1 && gem uninstall -I nsq-ruby --version 1.7.0

• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
• Kaniko Image (fully qualified with digest)

tar -cf - Dockerfile | gzip -9 | docker run   --interactive -v `pwd`/config.json:/kaniko/.docker/config.json:ro -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest   --context tar://stdin --destination qiutongs/kaniko-example:latest --force

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[acouvreur]

We copy directly our build sources from the CI into the container, some of our files vanished in the final image.
We replaced the default nginx landing page (index.html), but the default file was kept.

[gabyx]

This is likely to get fixed: in #2066
Which corrects some really strange bugs with deleting and adding files.

[aaron-prindle]

Can anyone in the thread here confirm if #2066 fixed this issue?

[ruiwen-zhao]

according to containerd/containerd#4659 (comment), this issue is fixed by #2066.

Closing.