Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add rebar_parallel pool, use in DAG scans #2768

Merged
merged 2 commits into from
May 15, 2023
Merged

Add rebar_parallel pool, use in DAG scans #2768

merged 2 commits into from
May 15, 2023

Commits on Feb 15, 2023

  1. Add rebar_parallel pool, use in DAG scans 

    This addresses erlang#2767 by creating
a pool mechanism in rebar_parallel that keeps as similar of an interface
as possible as the queue mechanism, with the one caveat that it allows
the asynchronous creation of tasks rather than requiring them all at
start time.

The mechanism is not tested super deeply, which is probably a mistake,
but I wanted to get a reviewable PR first.

The mechanism is also added to the rebar_compiler_dag module to cover
use cases that were handled by spawning an unbounded number of processes
before, which would cause problem with low file descriptors being
allocated and lots of files being used and open in parallel. The pool
mechanism puts an upper bound on processing but also on resource usage.

So this PR may also come with a performance regression, and if so we'd
want to override the default 1-per-scheduler pool options to use a lot
more and hit a middleground in performance vs. resource usage.
    @ferd
    ferd committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    235d579 View commit details
    Browse the repository at this point in the history

  2. Add types and docs to rebar_parallel

    @ferd
    ferd committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    4583c25 View commit details
    Browse the repository at this point in the history