cgroup v2 error - "sed: write error"

[gene-miller]

The 'dind' script assumes that the dockerd process will start with pid 1.
This is often not the case when building from the dind image and this causes an error "sed: write error" on systems that use cgroup v2.

Could I suggest checking the process ID of dockerd rather than assuming it is 1 in the dind script? Something like:

# cgroup v2: enable nesting
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
	# move the init process (PID 1) from the root group to the /init group,
	# otherwise writing subtree_control fails with EBUSY.
	mkdir -p /sys/fs/cgroup/init
	DOCKERDPID=$(/usr/bin/pgrep dockerd | wc -l)
	echo $DOCKERDPID > /sys/fs/cgroup/init/cgroup.procs
	# enable controllers
	sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers > /sys/fs/cgroup/cgroup.subtree_control
fi

[tianon]

Ah, looks like this code was introduced in moby/moby#41065 (this script comes from https://github.com/moby/moby/blob/6110ba3d7c8b83794a8d2e915410c11e7460e4b5/hack/dind).

Given that this script is used as an entrypoint, it will normally start before dockerd (it is intended to wrap the dockerd invocation), so I think instead of trying to look for DOCKERDPID or hard-coding 1, we should probably just be using $$, but I have to admit I don't understand the details / cgroupsv2 sufficiently to say for certain. It's probably worth opening this as an issue or PR on https://github.com/moby/moby to discuss more with the maintainers there.

[AkihiroSuda]

This has to be done with PID 1.

How do you get non-1 PID?

[tianon]

I can reproduce by just using --init on my Docker-in-Docker container on a cgroupsv2 host. 😬

$ docker pull docker:dind
dind: Pulling from library/docker
Digest: sha256:e0cef8e03463c7dde0613bb68a3fa211f4e3a12823b38f03f92bf330abaef3a9
Status: Image is up to date for docker:dind
docker.io/library/docker:dind
$ docker run -it --rm --init --privileged -e DOCKER_TLS_CERTDIR= docker:dind
sed: write error

[AkihiroSuda]

PR: moby/moby#42331

[tianon]

Should be fixed via docker-library/official-images#10087! Thank you @AkihiroSuda 😄

(30d7b9b)

[rousku]

I get a similar error with the following script. It takes some time for the issue to appear.

#!/bin/bash

while true; do
   DIND_CONTAINER_ID=$(docker run -t --privileged -d docker:26.1.2-dind)
   echo $DIND_CONTAINER_ID
   while ! docker exec "$DIND_CONTAINER_ID" docker info | grep "Server Version: 26.1.2"; do
   	sleep 1
   done
   docker stop $DIND_CONTAINER_ID
   docker rm $DIND_CONTAINER_ID
done

.
.
.
fc7502cfe87e48cfd464d4a5713f2efaf5e2b4341d5a13d5381c324ff80ec8df
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
 Server Version: 26.1.2
fc7502cfe87e48cfd464d4a5713f2efaf5e2b4341d5a13d5381c324ff80ec8df
fc7502cfe87e48cfd464d4a5713f2efaf5e2b4341d5a13d5381c324ff80ec8df
4940ab34359e57a661bc80bfa6aa9afa2de2cf9fc2b2609a0ec595044e3c314e
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
Error response from daemon: container 4940ab34359e57a661bc80bfa6aa9afa2de2cf9fc2b2609a0ec595044e3c314e is not running
Error response from daemon: container 4940ab34359e57a661bc80bfa6aa9afa2de2cf9fc2b2609a0ec595044e3c314e is not running
Error response from daemon: container 4940ab34359e57a661bc80bfa6aa9afa2de2cf9fc2b2609a0ec595044e3c314e is not running
Error response from daemon: container 4940ab34359e57a661bc80bfa6aa9afa2de2cf9fc2b2609a0ec595044e3c314e is not running
Error response from daemon: container 4940ab34359e57a661bc80bfa6aa9afa2de2cf9fc2b2609a0ec595044e3c314e is not running
.
.
.

$ docker logs 4940ab34359e57a661
Certificate request self-signature ok
subject=CN = docker:dind server
/certs/server/cert.pem: OK
Certificate request self-signature ok
subject=CN = docker:dind client
/certs/client/cert.pem: OK
cat: can't open '/proc/net/ip6_tables_names': No such file or directory
cat: can't open '/proc/net/arp_tables_names': No such file or directory
iptables v1.8.10 (nf_tables)
sed: write error

[tianon]

Yeah, I see this pretty regularly over in https://github.com/tianon/dockerfiles/actions?query=is%3Afailure (a non-trivial number of those failures are this exact error -- as far as I can tell, some kind of race condition between the shell and the kernel 😭).

sed: couldn't flush stdout: Device or resource busy