Skip to content

Commit

Permalink
feat(rosetta): JSII_ROSETTA_MAX_WORKER_COUNT limits max workers
Browse files Browse the repository at this point in the history
The current behavior of rosetta (extract) is to use a number of node worker
threads equal to half the number of CPU cores. On large CI/CD build hardware,
this may mean creating a huge number of worker threads which causes more
contention than benefit.

Introduce JSII_ROSETTA_MAX_WORKER_COUNT to limit the maximum number of workers
(defaults to 16).

See aws/aws-cdk#14389 for more motivation from the AWS CDK.
  • Loading branch information
njlynch committed Apr 27, 2021
1 parent 3e4c47e commit b72c7dd
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 1 deletion.
4 changes: 4 additions & 0 deletions packages/jsii-rosetta/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,3 +194,7 @@ Worker threads are enabled by default on NodeJS 12.x, and can be enabled on Node
```
$ node --experimental-worker /path/to/jsii-rosetta extract ...
```

If worker thread support is enabled, `jsii-rosetta` will use a number of workers equal to half the number of CPU cores,
up to a maximum of 16 workers. This default maximum can be overridden by setting the `JSII_ROSETTA_MAX_WORKER_COUNT`
environment variable.
6 changes: 5 additions & 1 deletion packages/jsii-rosetta/lib/commands/extract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,11 @@ async function workerBasedTranslateAll(
): Promise<TranslateAllResult> {
// Use about half the advertised cores because hyperthreading doesn't seem to help that
// much (on my machine, using more than half the cores actually makes it slower).
const N = Math.max(1, Math.ceil(os.cpus().length / 2));
// Cap to a reasonable top-level limit to prevent thrash on machines with many, many cores.
const maxWorkers = parseInt(
process.env.JSII_ROSETTA_MAX_WORKER_COUNT ?? '16',
);
const N = Math.min(maxWorkers, Math.max(1, Math.ceil(os.cpus().length / 2)));
const snippetArr = Array.from(snippets);
const groups = divideEvenly(N, snippetArr);
logging.info(
Expand Down

0 comments on commit b72c7dd

Please sign in to comment.