Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nix 2.24.4 has an increase of CURLM_UNRECOVERABLE_POLL errors #11387

Closed
grahamc opened this issue Aug 30, 2024 · 1 comment
Closed

Nix 2.24.4 has an increase of CURLM_UNRECOVERABLE_POLL errors #11387

grahamc opened this issue Aug 30, 2024 · 1 comment
Labels
bug

Comments

@grahamc
Copy link
Member

grahamc commented Aug 30, 2024 •
edited
Loading

Describe the bug

In our testing and monitoring at Determinate Systems, we've identified an increase in CURLM_UNRECOVERABLE_POLL errors while substituting. This is especially prevalent when a derivation has a large number of direct dependencies that are all substitutable, where Nix is able to substitute dozens to hundreds of paths at once.

We believe this problem is the same identified by curl/curl#14414. This is corroborated by our internal data:

  • We didn't see this issue at all on 2.23.3, which uses curl 8.4.0.
  • We do see this issue on 2.24.4, which uses curl 8.7.1.

This issue seems to be mitigated by increasing the daemon's NOFILE limit from the Nix default of 1,048,576 to something larger, for example 536,870,912. However, this is not recommended for performance reasons.

Note: we have primarily observed this on macOS.

Steps To Reproduce

  1. Install a Nix version greater than or equal to 2.24.0
  2. Set the NOFILE limit to 1,048,576
  3. Evaluate and build a derivation with hundreds of inputs that can all be immediately substituted
  4. Observe the error:
unexpected error in download thread: error: unexpected error from curl_multi_wait(): Unrecoverable error in select/poll
@grahamc grahamc added the bug label Aug 30, 2024
grahamc added a commit to DeterminateSystems/nix-installer that referenced this issue Aug 30, 2024
@grahamc
Raise the NOFILE limit to 512 * 1024 * 1024 after hitting limits talk…
d1b213b 
…ing to caches

See: NixOS/nix#11387
@grahamc grahamc mentioned this issue Aug 30, 2024
Merged
6 tasks
grahamc added a commit to DeterminateSystems/nix-installer that referenced this issue Aug 30, 2024
@grahamc
Update and unify limits across systemd and macOS. (#1126)
88077e7 
* Raise the NOFILE limit to 512 * 1024 * 1024 after hitting limits talking to caches

See: NixOS/nix#11387

* Set the default stack limit to 64M

Amazon Linux 2023 and likely CentOS / RedHat sets the default stack limit to 10M, and is generally effective at building nixpkgs. One notable exception is Nix, which typically sets it stack size to 64M. When it could not do that, a test around the stack overflow behavior failed. This was fixed in NixOS/nix#10903 to use less stack. However, since Nix typically can use a 64M stack we set that as the default max.

We should not treat this value as magically correct. If this value causes problems, we should freely raise it. It is possible infinity is the right answer. 64M is attempting to be a "good guess" at a universally applicable good start.

For future readers, this bug report to systemd regarding LimitSTACK and DefaultLimitSTACK might be useful: systemd/systemd#34193.

* Echo TasksMax from the systemd unit to the macOS plist

---

Also:

* Fixup the unused init parameter to ProvisionDeterminateNixd

* Fixup clippy warnings around borrows

* Delete dead code in configure_init_service
grahamc added a commit to DeterminateSystems/nix-installer that referenced this issue Aug 30, 2024
@grahamc
Go back to Nix 2.23.3 due to an upstream regression until we can dig …
e7a976f 
…in further

See: NixOS/nix#11387
@grahamc grahamc mentioned this issue Aug 30, 2024
Merged
6 tasks
cole-h pushed a commit to DeterminateSystems/nix-installer that referenced this issue Aug 30, 2024
@grahamc
Update flake inputs, downgrade Nix to 2.23.3 due to curl issues (#1129)
612ae86 
* flake.lock: Update

Flake lock file updates:

• Updated input 'determinate':
    'https://api.flakehub.com/f/pinned/DeterminateSystems/determinate/0.1.83%2Brev-609f242535bbd9e02024904aa85dc76848ee844d/01917d1e-ee3e-77ad-a117-9c214d4dcde7/source.tar.gz?narHash=sha256-eNRmQeWgf9IDI4SFgHibqt6TPaihLqPP%2Bh3w%2B%2B3PfH4%3D' (2024-08-23)
  → 'https://api.flakehub.com/f/pinned/DeterminateSystems/determinate/0.1.89%2Brev-54bcee31752428b7a69200be76e7c357723ae2de/0191a4ff-ffd4-7774-abcf-bf93c97c71c8/source.tar.gz?narHash=sha256-hU7e8tuhxi3jQxJXsqaG%2BzhhNodV3oVzp9FxzOnuEbY%3D' (2024-08-30)
• Updated input 'determinate/determinate-nixd-aarch64-darwin':
    'https://install.determinate.systems/determinate-nixd/rev/21cb4d451a3d3a9ea72fb5a25c691eb4438d210a/macOS?narHash=sha256-IKnMJtg%2BAxXg5H2/hSJgoHxo42LqDSJlxzpIyHR1lnU%3D'
  → 'https://install.determinate.systems/determinate-nixd/rev/fb88a079cf330e8dfe20f4426a36ee663d7bb47e/macOS?narHash=sha256-PE5iOUHttLNVnoW/HU2CJbIxDhwvpqM7ZehNxo8G45Q%3D'
• Updated input 'determinate/determinate-nixd-aarch64-linux':
    'https://install.determinate.systems/determinate-nixd/rev/21cb4d451a3d3a9ea72fb5a25c691eb4438d210a/aarch64-linux?narHash=sha256-zPzIinp47RCpeMZWiDW3I8P1BDfE5hyJgSvbvoBJ%2Bcg%3D'
  → 'https://install.determinate.systems/determinate-nixd/rev/fb88a079cf330e8dfe20f4426a36ee663d7bb47e/aarch64-linux?narHash=sha256-u1RycvQDu9VevkjHlfiNvbk566em52hDvq%2BKoLiY7Kg%3D'
• Updated input 'determinate/determinate-nixd-x86_64-linux':
    'https://install.determinate.systems/determinate-nixd/rev/21cb4d451a3d3a9ea72fb5a25c691eb4438d210a/x86_64-linux?narHash=sha256-4EkN/ImFB22m%2BFmJ2Rb5Y/mStjOqJWsSeIJs9fsG0Vg%3D'
  → 'https://install.determinate.systems/determinate-nixd/rev/fb88a079cf330e8dfe20f4426a36ee663d7bb47e/x86_64-linux?narHash=sha256-x4au1LaMr/SqiFcbt1GEq1QAlIf9AB9K0T/AH3AvrjY%3D'
• Updated input 'nix':
    'https://api.flakehub.com/f/pinned/DeterminateSystems/nix/2.24.4/01917fb6-9673-783a-9c79-5a46636fe80d/source.tar.gz?narHash=sha256-mSYGU5vXXrrfSAailue%2BUgWNxfnL7AOVZ4RCdPK2mtM%3D' (2024-08-23)
  → 'https://api.flakehub.com/f/pinned/DeterminateSystems/nix/2.23.3/019097ec-5f84-7a24-9af5-79a2dfa6fe73/source.tar.gz?narHash=sha256-l8Q5/8DwzkW2FgT9Iicxtzxj/MMNE2YlTKWlCV5ybko%3D' (2024-07-09)
• Updated input 'nix/nix':
    'https://api.flakehub.com/f/pinned/NixOS/nix/2.24.4/01917f97-644e-756d-93f3-051d3e3b9817/source.tar.gz?narHash=sha256-oYu/9u8ht34JOTV%2BG/l3CCFJokPiUA2D8CiLZFX61PA%3D' (2024-08-23)
  → 'https://api.flakehub.com/f/pinned/NixOS/nix/2.23.3/0190936a-a531-7743-88ed-025ecd4d0835/source.tar.gz?narHash=sha256-lAoLGVIhRFrfgv7wcyduEkyc83QKrtsfsq4of%2BWrBeg%3D' (2024-07-05)
• Updated input 'nix/nix/flake-compat':
    'github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33?narHash=sha256-kvjfFW7WAETZlt09AgDn1MrtKzP7t90Vf7vypd3OL1U%3D' (2023-10-04)
  → 'github:edolstra/flake-compat/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9?narHash=sha256-4gtG9iQuiKITOjNQQeQIpoIB6b16fm%2B504Ch3sNKLd8%3D' (2023-01-17)
• Updated input 'nix/nix/flake-parts':
    'github:hercules-ci/flake-parts/9227223f6d922fee3c7b190b2cc238a99527bbb7?narHash=sha256-pQMhCCHyQGRzdfAkdJ4cIWiw%2BJNuWsTX7f0ZYSyz0VY%3D' (2024-07-03)
  → 'github:hercules-ci/flake-parts/9126214d0a59633752a136528f5f3b9aa8565b7d?narHash=sha256-sB4SWl2lX95bExY2gMFG5HIzvva5AVMJd4Igm%2BGpZNw%3D' (2024-04-01)
• Removed input 'nix/nix/git-hooks-nix'
• Removed input 'nix/nix/git-hooks-nix/flake-compat'
• Removed input 'nix/nix/git-hooks-nix/gitignore'
• Removed input 'nix/nix/git-hooks-nix/nixpkgs'
• Removed input 'nix/nix/git-hooks-nix/nixpkgs-stable'
• Updated input 'nix/nix/libgit2':
    'github:libgit2/libgit2/36f7e21ad757a3dacc58cf7944329da6bc1d6e96?narHash=sha256-J2rCxTecyLbbDdsyBWn9w7r3pbKRMkI9E7RvRgAqBdY%3D' (2024-05-16)
  → 'github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D' (2023-10-18)
• Updated input 'nix/nix/nixpkgs':
    'github:NixOS/nixpkgs/c3d4ac725177c030b1e289015989da2ad9d56af0?narHash=sha256-sqLwJcHYeWLOeP/XoLwAtYjr01TISlkOfz%2BNG82pbdg%3D' (2024-08-15)
  → 'github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D' (2024-02-28)
• Removed input 'nix/nix/nixpkgs-23-11'
• Added input 'nix/nix/pre-commit-hooks':
    'github:cachix/pre-commit-hooks.nix/40e6053ecb65fcbf12863338a6dcefb3f55f1bf8?narHash=sha256-nMirxrGteNAl9sWiOhoN5tIHyjBbVi5e2tgZUgZlK3Y%3D' (2024-04-12)
• Added input 'nix/nix/pre-commit-hooks/flake-compat':
    follows 'nix/nix'
• Added input 'nix/nix/pre-commit-hooks/flake-utils':
    'github:numtide/flake-utils/5aed5285a952e0b949eb3ba02c12fa4fcfef535f?narHash=sha256-nuEHfE/LcWyuSWnS8t12N1wc105Qtau%2B/OdUAjtQ0rA%3D' (2022-11-02)
• Added input 'nix/nix/pre-commit-hooks/gitignore':
    follows 'nix/nix'
• Added input 'nix/nix/pre-commit-hooks/nixpkgs':
    follows 'nix/nix/nixpkgs'
• Added input 'nix/nix/pre-commit-hooks/nixpkgs-stable':
    follows 'nix/nix/nixpkgs'
• Updated input 'nix/nixpkgs':
    'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2405.634339%2Brev-797f7dc49e0bc7fab4b57c021cdf68f595e47841/01917ea1-8ce4-7d71-a601-f943a160def2/source.tar.gz?narHash=sha256-Qb9MhKBUTCfWg/wqqaxt89Xfi6qTD3XpTzQ9eXi3JmE%3D' (2024-08-22)
  → 'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.2405.631440%2Brev-2819fffa7fa42156680f0d282c60d81e8fb185b7/0190034c-678d-7039-b45c-fa38168f2500/source.tar.gz?narHash=sha256-mJi4/gjiwQlSaxjA6AusXBN/6rQRaPCycR7bd8fydnQ%3D' (2024-06-09)
• Updated input 'nixpkgs':
    'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.1.669741%2Brev-c374d94f1536013ca8e92341b540eba4c22f9c62/019178de-6006-7f2e-8b92-4b3b936604b8/source.tar.gz?narHash=sha256-Z/ELQhrSd7bMzTO8r7NZgi9g5emh%2BaRKoCdaAv5fiO0%3D' (2024-08-21)
  → 'https://api.flakehub.com/f/pinned/NixOS/nixpkgs/0.1.672439%2Brev-71e91c409d1e654808b2621f28a327acfdad8dc2/01919c14-b63e-7736-a9e9-48bee9f65f2b/source.tar.gz?narHash=sha256-GnR7/ibgIH1vhoy8cYdmXE6iyZqKqFxQSVkFgosBh6w%3D' (2024-08-28)

* Go back to Nix 2.23.3 due to an upstream regression until we can dig in further

See: NixOS/nix#11387
@edolstra edolstra mentioned this issue Sep 2, 2024
Merged
@grahamc
Copy link
Member Author

grahamc commented Sep 3, 2024

I think this is solved in #11402

@grahamc grahamc closed this as completed Sep 3, 2024
emilazy added a commit to emilazy/nixpkgs that referenced this issue Sep 5, 2024
@emilazy
nixVersions.nix_2_24: 2.24.4 -> 2.24.5
06587a2 
Fixes an [annoying macOS regression].

[annoying macOS regression]: NixOS/nix#11387

Diff: NixOS/nix@2.24.4...2.24.5
@emilazy emilazy mentioned this issue Sep 5, 2024
Merged
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@grahamc