Recursive chown is really slow

[katsar0v]

• This is a bug report
• [] This is a feature request
• I searched existing issues before opening this one (found similar issue, which was closed, but the problem still persist with me)

Expected behavior

I want to build an image with simple wordpress and theme installation (with nginx, mysql and so on).

Actual behavior

I am building an image with WordPress and own templates. In order for everything to work correctly, I need to set chown www-data:www-data /var/www/html/ -R. Unfortunately this takes long time, at least 5 minutes most of the time.

Steps to reproduce the behavior

Step 91/95 : WORKDIR /var/www/html/
 ---> Running in 3bbf283b7604
Removing intermediate container 3bbf283b7604
 ---> abe7aa3891b3
Step 92/95 : RUN chown www-data:www-data . -R

This is where this hangs for at least 5+ minutes.

Output of docker version:

Docker version 18.06.0-ce, build 0ffa825

Output of docker info:

Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 2246
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-46-generic
Operating System: Ubuntu 17.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.688GiB
Name: katsarov-ThinkPad-T470s
ID: R5S6:67UI:QUZK:YBZS:VXLQ:YEFK:SEGU:7EAO:WTUZ:63R5:V3UX:NEJT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

None, running on a fresh ubuntu machine.

[cpuguy83]

This is likely because every file has to be copied up on a CoW filesystem. This also means data is duplicated in the image.

How many/what size are these files? Can you add them in with the correct uid/gid instead of chowning in the build? (Note there is a "--chown" flag on the COPY instruction)

[katsar0v]

Hi, thank for your reply. I will try the --chown flag. The chown is executed on a wordpress installation (not big, but many files). But what if the files come from git, not from host machine?

[ZelphirKaltstahl]

I noticed the --chown COPY argument does not seem to support variables defined inside the Dockerfile. I presume it is because those are only defined at build time?

This is important when you do not know the user ahead of time and have it stored in a variable or when you do not wish to hardcode the user into many places inside the Dockerfile, so that you do not need to change so many places when changing the user / username.

Should this be a new issue?

[Grant1219]

I also started running into the issue today while debugging a container that is running chown -R over about 40k files and about 400MB of files. It takes roughly 12 minutes to complete.

The more interesting part is that this same container takes only 2 minutes to start up on an Ubuntu 14.04 server (4.4 kernel), but with Ubuntu 16.04 (4.15 kernel) the time required increases significantly. Docker versions on both servers are the same 18.03.1-ce.

EDIT:
After more digging, I discovered the underlying storage driver changed from "aufs" to "overlay2" so I believe the slowdown must be because of that change.

[cpuguy83]

The good news, in the new kernel overlay does a meta-data only copy for this instead of copying the whole file.

[Grant1219]

@cpuguy83 Which version will that be included in?

[cpuguy83]

@Grant1219 Looks like 4.19 -- https://cateee.net/lkddb/web-lkddb/OVERLAY_FS_METACOPY.html
You can set the default as a kernel config as well as set a mount option.

[katsar0v]

Does switching to a newer kernel really help in this case?

[cpuguy83]

If you switch and turn on meta-only copy by default in the kernel config, I think it would make a huge difference, but I have not tested yet.

Also keep in mind this is only available in 4.19 which is still in the RC phase.

[cpuguy83]

The smaller the sizes and greater number of files the less effective it would be, I suspect.
But I don't see how chown could improve beyond that change.

[kavehv]

I just migrated from my Ubuntu 14.04 VM to an 18.04 VM and attempted to bring up the same docker image in a container on the same version of docker (18.06.1). We also do a similar chown in our entrypoint. It's FAR slow in 18.04 than it was in 14.04. Both are set up to use overlay2.

[katsar0v]

Okay, this seems to be real issue and not a small one. @kavehv you say it gets worse on 18.04, can you also compare the kernel versions?

[kavehv]

14.04:

$ docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 147
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
Kernel Version: 4.4.0-79-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 10.35GiB
Name: kaveh-ubuntu
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

18.04:

$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 33
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-34-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 10.47GiB
Name: kaveh-ubuntu
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[kavehv]

Definitely seems like a consistent problem with 18.04 and overlay2. We see it on more than one system here. Is this the right project to report this issue?

[cpuguy83]

Probably best to report to Canonical in this case.

[kavehv]

FWIW, the same thing appears to happen on 16.04 as well (similar 4.15 kernel version):

$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-29-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 9.442GiB
Name: kaveh-ubuntu
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Is it ubuntu or the kernel? Does someone more familiar with docker-ce need to file a bug?

[kavehv]

Also, modifying /etc/docker/daemon.json to specify devicemapper as your storage driver (with an appropriate base size if necessary) seems to be the way we'll have to work around this issue. It reduces this operation from 3 minutes back down to about 3 seconds.

[Grant1219]

I have resolved this issue for my specific case (Ubuntu 16.04 and Docker 18.03). It turned out to be the change of the default IO scheduler from "deadline" on 14.04 to "cfq" on 16.04. I did some benchmarking outside of Docker (and inside) and found the same slowness for any kind of write operation.

Although it may not completely resolve the performance differences, it may be worth checking which scheduling algorithm is used for IO on the machines you are seeing the issues on.

[kavehv]

I have resolved this issue for my specific case (Ubuntu 16.04 and Docker 18.03). It turned out to be the change of the default IO scheduler from "deadline" on 14.04 to "cfq" on 16.04. I did some benchmarking outside of Docker (and inside) and found the same slowness for any kind of write operation.

Although it may not completely resolve the performance differences, it may be worth checking which scheduling algorithm is used for IO on the machines you are seeing the issues on.

This is a good find, but are you sure this wasn't changed in 14.04?
https://askubuntu.com/questions/784442/why-does-ubuntu-16-04-set-all-drive-io-schedulers-to-deadline

[katsar0v]

Any updates from the devs of the repo? If this repo is not the correct when, where should I post this. I see this is becoming bigger problem with docker?

[katsar0v]

Here is a full Dockerfile example (with WordPress plugins) for testing the chmod performance:

FROM ubuntu
RUN apt-get update && apt-get install -y wget unzip
############
# ENV VARS #
############
ARG DEBIAN_FRONTEND="noninteractive"
ENV SHELL /bin/bash
# user and group
ENV NON_PRIVILEGED_USER="test"
ARG NON_PRIVILEGED_USER_PASSWORD="test"
ENV NON_PRIVILEGED_USER_GROUP="test"

#################
# ADDING A USER #
#################
# For more info please check useradd --help or man useradd.
RUN groupadd -r $NON_PRIVILEGED_USER_GROUP -g 1000 \
 && useradd \
    --uid 1000 \
    --system \
    --gid $NON_PRIVILEGED_USER_GROUP \
    --create-home \
    --home-dir /home/$NON_PRIVILEGED_USER/ \
    --shell /bin/bash \
    --comment "non-privileged user" \
    $NON_PRIVILEGED_USER \
 && chmod 755 /home/$NON_PRIVILEGED_USER/ \
 && echo "$NON_PRIVILEGED_USER:$NON_PRIVILEGED_USER_PASSWORD" | chpasswd

RUN mkdir /tmp/fold1/
WORKDIR /tmp/fold1/
ARG MAILPOET_DOWNLOAD_URL="https://downloads.wordpress.org/plugin/mailpoet.3.11.0.zip"
ARG USER_ROLE_EDITOR_DOWNLOAD_URL="https://downloads.wordpress.org/plugin/user-role-editor.4.46.zip"
ARG CMS_PAGE_TREE_VIEW_DOWNLOAD_URL="https://downloads.wordpress.org/plugin/cms-tree-page-view.1.5.zip"
RUN wget --output-document mailpoet.zip --quiet ${MAILPOET_DOWNLOAD_URL} && \
    wget --output-document user_role_editor.zip --quiet ${USER_ROLE_EDITOR_DOWNLOAD_URL} && \
    wget --output-document cms_tree_view.zip --quiet ${CMS_PAGE_TREE_VIEW_DOWNLOAD_URL} && \
    unzip mailpoet.zip && rm mailpoet.zip && \
    unzip user_role_editor.zip && rm user_role_editor.zip && \
    unzip cms_tree_view.zip && rm cms_tree_view.zip
RUN chown test:test . -R
CMD ["bash"]

[OhMyMndy]

This problem does not only occur on Ubuntu 18.04, I have the same problem on Fedora 28.
Changing IO scheduler to cfq or deadline did nothing for me.

System information:
/var/lib/docker is on a separate partition (ext4 7200RPM disk)
Kernel 4.17.19-200.fc28.x86_64
Docker version 18.06.1-ce, build e68fc7a
Storage driver overlay2 on extfs

Minimal test Dockerfile with Centos
The Symfony repository is 177MB when cloned
it takes over 4 minutes to chown on overlay2
it takes over 0.5 seconds to chown on devicemapper

FROM centos:latest
RUN yum install git -y
RUN useradd -m test
RUN cd /home/test && git clone https://github.com/symfony/symfony.git
RUN time chown -R test:test /home/test

[jpic]

An alternative to COPY --chown, is to just install all as root, then drop priv with USER, the user will need to write only in volumes and tempfiles i suppose.

[katsar0v]

An alternative to COPY --chown, is to just install all as root, then drop priv with USER, the user will need to write only in volumes and tempfiles i suppose.

This only improves partially the performance.

[cpuguy83]

FYI, there appears to be some other grumblings wrt overlayfs performance since 4.15. Has anyone compared chown performance against older versions of the kernel?

…

On Mon, Oct 15, 2018 at 8:44 AM Kristiyan ***@***.***> wrote: An alternative to COPY --chown, is to just install all as root, then drop priv with USER, the user will need to write only in volumes and tempfiles i suppose. This only improves partially the performance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

-- - Brian Goff

[KernelMadness]

Experiencing the same slowdown in 18.04(kernel 4.15) vs 16.04(kernel 4.4).
4.4 is a way faster, like 5-10 times

[jpic]

This only improves partially the performance.

My suggestion completely removes the performance bottleneck, I don't understand how i can be a "partial" solution, can you elaborate please ?

Not to mention, how ugly is a chown call inside a Dockerfile as that's a matter of taste (which not everybody will share).

Thanks ;)

[katsar0v]

This only improves partially the performance.

My suggestion completely removes the performance bottleneck, I don't understand how i can be a "partial" solution, can you elaborate please ?

Not to mention, how ugly is a chown call inside a Dockerfile as that's a matter of taste (which not everybody will share).

Thanks ;)

Removing the performance bottleneck would mean for me that chown inside dockerfile / docker container takes exactly the same time as on host machine (several ms) which is definitely not the case.
chown in a RUN command takes approx. up to 2 minutes for a big directory with many files. When copying with --chown it takes about ~10-30 seconds which is really better but still not comparable to 0.001 seconds (benchmark on host machine)

[cpuguy83]

Unfortunately the bottleneck comes from overlayfs in the kernel. It would probably be best to report the issue to your distro provider to get more eyes on the issue.

As for COPY --chown taking a long time, we should look at the cause of this. There will be some overhead here just because we are copying to overlayfs, but 10-30s compared to sub-second seems a bit wrong. How large is this copy? How many files? How long does it take if you use DOCKER_BUILDKIT=1 docker build <args> with an experimental docker daemon enabled?

Without buildkit, I believe COPY creates a tar of the items to be copied and untars at the destination... this could be part of the overhead but hard to say without some comparisons.

[kavehv]

An alternative to COPY --chown, is to just install all as root, then drop priv with USER, the user will need to write only in volumes and tempfiles i suppose.

This is not a viable workaround for those of us running into this issue in the docker entrypoint. A good number of us who have reported this issue use a recursive chown in the entrypoint because we need to adapt some things in the image to the UID/GID of the user running the container. There's no way [that I'm aware of] to solve for this in the Dockerfile.

[scalavision]

I can also confirm this, simple chown of some dirs, takes at least 15 mins.

[theamanbhargava]

Any new about this? I have my builds stuck for 20 minutes just because of this issue, it worth mentioning that we are using docker-in-docker for our build

Instead:
COPY /src /dest
RUN chown user /dest
Use:
COPY --chown=user:group /src /dest

This solved my issue, thanks @Belanchuk

Does this work with env vars?

Yes it does for me

[ap-wtioit]

This issue not only affects chown (can be workarrounded with COPY --chown=) but also chmod for which no workarround exists (AFAIK).

[theamanbhargava]

I see. This issue needs fixing soon!

…

On Mon, 18 Oct 2021 at 2:40 PM, Andreas Perhab ***@***.***> wrote: This issue not affects chown (can be workarrounded with COPY --chown=) but also chmod for which no workarround exists (AFAIK). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#388 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRBZHNW3R6CW27FPDKP6KDUHPP75ANCNFSM4FOGNQ6A> .

[Pi-George]

Sending build context to Docker daemon  3.533GB
Step 1/4 : ARG FROM
 --->
Step 2/4 : FROM $FROM as production
 ---> ed6e6e960cc6
Step 3/4 : COPY ./ /srv/www/
 ---> 6f7b55f10623
Step 4/4 : RUN chown -R root:root /srv/www/
 ---> Running in 1c26e11d5632

I gave it two hours and it still hasn't finished

[benken-parasoft]

Before running chown -R, I temporarily replace /var/log/lastlog and /var/log/faillog with symlinks to /dev/null. I believe this has to do with how docker handles sparse files. This solves the problem for me but wondering if anyone has a better workaround.

[Pi-George]

Using --chown on the COPY command solved it for me

[ZelphirKaltstahl]

Using --chown on the COPY command solved it for me

That's only half a solution, because it does not work with environment variables as user names.

[cpuguy83]

@ZelphirKaltstahl

The newest Dockerfile parser handles variables on flags.

FROM busybox
ENV THE_USER=nobody
COPY --chown=${THE_USER} foo /foo
RUN ls -lh /foo

docker buildx build --no-cache --progress=plain /tmp/test                                                                                    
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 121B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 0.4s

#4 [1/3] FROM docker.io/library/busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678
#4 resolve docker.io/library/busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678 0.0s done
#4 CACHED

#5 [internal] load build context
#5 transferring context: 22B done
#5 DONE 0.0s

#6 [2/3] COPY --chown=nobody foo /foo
#6 DONE 0.0s

#7 [3/3] RUN ls -lh /foo
#7 0.115 -rw-rw-r--    1 nobody   nobody         0 Feb  2 23:31 /foo
#7 DONE 0.1s
``

[benken-parasoft]

In my case, I just need to run chown -R on a directory from a base image. I'm not copying anything and therefore wouldn't be running COPY --chown, right?

[andorfermichael]

still no solution here?

[jessesimpson36]

When I was dealing with this issue, upgrading past linux kernel 4.19 resolved my issue. Versions past 4.19 have a patch that makes the overlay2 filesystem use metadata-only copyup during chown rather than a regular copy-up which is a significant performance boost.

I think this github issue is perhaps not specific enough in terms of whether that speed increase was significant enough to close the issue or keep open. There are other changes you can make as well like COPY --chown instead of chmod -R, which some people may not be able to apply to their own situation. Also perhaps changing the docker daemon's configuration for the type of filesystem used.

Hope that helps @andorfermichael . I ran into this issue a couple years ago and just kept following it. I'm not a docker/for-linux developer or anything.

[NoamNol]

Yes COPY --chown is much faster for me, and to reduce space I use it with multi-stage builds.

To build my nodejs app, Instead of:

FROM node:16-alpine AS deps
RUN apk add --no-cache libc6-compat
RUN npm install -g npm@latest
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

FROM node:16-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .

RUN npm run build

# This is slow!
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser

CMD ["npm", "run", "start"]

I did:

FROM node:16-alpine AS deps
RUN apk add --no-cache libc6-compat
RUN npm install -g npm@latest
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

FROM node:16-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
# (ignore files in .dockerignore)
COPY . .

RUN npm run build

FROM node:16-alpine AS runner
WORKDIR /app

RUN addgroup --system --gid 1001 appgroup
RUN adduser --system --uid 1001 appuser

# Faster!
COPY --from=builder --chown=appuser:appgroup /app ./

USER appuser

CMD ["npm", "run", "start"]

[sebastianvitterso]

I had the problem of RUN chown -r user /app taking a while (too long for me to bother waiting). The issue turned out to be that my node_modules folder was being copied and attempted chowned, which was completely unnecessary.

The fix: Add a .dockerignore like:

node_modules