Rootless --cpus results in Permission Denied

[jam01]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Executing a simple podman run -it --rm -p 9000:9000 --memory 1G --cpus 1 localhost/bench-camel:latest results in

Error: opening file `cpu.max` for writing: Permission denied: OCI runtime permission denied error

Executing under sudo works correctly.

This issue seemed related containers/crun#489 but if I remove the --cpus flag the memory limit is applied correctly, so it's something specific to cpus.

Steps to reproduce the issue:

1. Trying to limit cpus with --cpus

Describe the results you received:
Permissions denied error

Describe the results you expected:
Successfully limit cpus

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14.9
Built:        Wed Sep 30 14:31:11 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: jam01.jam01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.12-200.fc32.x86_64
  linkmode: dynamic
  memFree: 1624256512
  memTotal: 16672882688
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 8246571008
  swapTotal: 8350855168
  uptime: 112h 5m 29.99s (Approximately 4.67 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
  - registry.redhat.io
store:
  configFile: /home/jam01/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/jam01/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 12
  runRoot: /run/user/1000/run
  volumePath: /home/jam01/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601494271
  BuiltTime: Wed Sep 30 14:31:11 2020
  GitCommit: ""
  GoVersion: go1.14.9
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-7.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No, and yes.

Additional environment details (AWS, VirtualBox, physical, etc.):
na

[rhatdan]

@giuseppe PTAL
I think this might be something to do with systemd setup of cgroups?

I have verified this on my system as well.
$ podman run --rm --cpus 1 alpine echo hello
Error: opening file cpu.max for writing: Permission denied: OCI runtime permission denied error

[giuseppe]

the cpu controller is not enabled by default for unprivileged users.

To verify it, try running $ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.controllers. What is the output?

Try creating a file /etc/systemd/system/user@.service.d/delegate.conf with the content:

[Service]
Delegate=memory pids cpu io

and restart your user session

[jam01]

cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.controllers
memory pids

after creating the delegate.conf file

cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/cgroup.controllers
cpu io memory pids

success:

podman run --rm --cpus 1 alpine echo hello
hello

What are the implications of adding this config file? I'm guessing there's a reason why it's not setup by default.

[giuseppe]

the reason for not enabling it by default is that the cpu controller used to be expensive, so systemd avoided to expose it to unprivileged users. AFAIK, that is not true anymore in newer kernels, so it should be fine to allow it

[jam01]

Made a quick search and found this https://wiki.archlinux.org/index.php/cgroups#User_Delegation

Not sure how it differs from the delegate.conf file, but I couldn't find info on that one after a (very quick) search. I can make a PR to the Troubleshooting Guide with some of this info, if you'd like?

[rhatdan]

@giuseppe Could you open an issue/bugzilla with systemd to enable this by default in fedora 33/34?

[giuseppe]

@keszybz is there any reason why it is not done yet?

[keszybz]

I don't think there's any particular reason, except for history as mentioned above. Do we have something from the kernel side that we could link to to justify that this is not expensive anymore?

[giuseppe]

Do we have something from the kernel side that we could link to to justify that this is not expensive anymore?

from a quick look, I cannot find anything in the git history

[keszybz]

@htejun, @poettering ?

[xordspar0]

FWIW, I can't reproduce this on the Fedora 33 beta.

[htejun]

Unfortunately, it is still expensive with increasing nesting levels. Didn't really improve much. I don't know whether that justifies user-disturbing behavior differences tho. For context switch sensitive workloads, we see the overhead being easily above a percent of total cpu time per level. Whether that's prohibitive or not depends on the specific use case.

[keszybz]

Thanks @htejun. In case of rootless podmans, we have fairly deep nesting levels, e.g. /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-ea238e8a-d779-4795-a6ce-e32fa68ccf53.scope. In addition, we would automatically enable accounting for sibling cgroups, i.e. a lot of the user session. So overall, this would be fairly expensive.

But we could enable Delegate=cpu (as per #7959 (comment)). This will enable the cpu controller for /user.slice, /user.slice/user-nnn.slice, /user.slice/user-nnn.slice/user@nnn.service, /system.slice, /machine.slice (but not nested units, until somebody requests it, i.e. e.g. until podman is started). It would make podman work better. I'm not sure if it's worth the overhead though.

[rhatdan]

It seems to me that we should document in the man page and perhaps make the error more helpful, to point them at turning on this nob. Only an extremely small number of users will ever run rootless containers with this type of setting, so the majority of users in the world should not pay the penalty.

[xordspar0]

Where do you think documentation like that should live? This effects several CPU-related flags, so I would maybe add a quick note under --cpus, --cpu-quota, etc. I don't see a troubleshooting section in the man pages. Maybe an example in podman-run(1), something like "setting CPU limits in rootless mode"?

[rhatdan]

Yes this should definitely be in the podman-run/podman-start man pages and should also be covered in troubleshoot.md.
I would just add a rootless.

I think adding a comment about limitations on each one of the options pointing at the explanation in a new troubleshooting section would work well in the man pages.

[xordspar0]

Yes this should definitely be in the podman-run/podman-start man pages and should also be covered in troubleshoot.md.

Done in #8134.

I would just add a rootless.

Sorry, I'm not sure I understand. Did you mean add an item to rootless.md? If so, my PR doesn't have that yet. We can discuss over there. 👉

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Man page and troubleshooting updated. Closing. Reopen if I am mistaken.