Emscripten fails on Node.js with PThreads and ES6 exports

[LTLA]

tl;dr Compiling with ES6, PThreads and Node options produces a not-quite-right *.worker.js file.

I have some C++ code that uses PThreads (greatly simplified below, let's call it dummy.cpp):

#include <thread>
#include <vector>
#include <emscripten/bind.h>

double executor(int nthreads) {
    std::vector<std::thread> workers;
    workers.reserve(nthreads);

    int first = 0;
    std::vector<double> output(nthreads);
    for (int w = 0; w < nthreads; ++w) {
        workers.emplace_back([&](int w) -> void { output[w] = w; }, w);
    }

    for (int w = 0; w < nthreads; ++w) {
        workers[w].join();
    }

    double accumulated = 0;
    for (auto o : output) { accumulated += o; }
    return accumulated;
}

EMSCRIPTEN_BINDINGS(xxxx) {
    emscripten::function("executor", &executor);
}

I compile this for Node.js with ES6 export, using Emscripten version 3.1.31 (e883361):

emcc -sUSE_PTHREADS=1 -sPTHREAD_POOL_SIZE=4 -sENVIRONMENT=node -sEXPORT_NAME=loadme --bind -sEXPORT_ES6 dummy.cpp -o dummy.js

I create the following script (test.js) - I also add a package.json with "type": "module" to treat it as ES6:

import loadme from "./dummy.js";

// Initializing this.
let module = await loadme();
console.log(module.executor(5));

And then run it (using Node v16.19.0), which throws the following error:

$ node --experimental-vm-modules test.js
worker sent an error! undefined:undefined: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and '/home/luna/Code/js/node-es6-pthreads/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.

file:///home/luna/Code/js/node-es6-pthreads/dummy.js:169
      throw ex;
      ^
ReferenceError [Error]: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and '/home/luna/Code/js/node-es6-pthreads/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
    at file:///home/luna/Code/js/node-es6-pthreads/dummy.worker.js:20:27
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:530:24)
    at async loadESM (node:internal/process/esm_loader:91:5)
    at async handleMainPromise (node:internal/modules/run_main:65:12)

And indeed, if I look at dummy.worker.js, there are a few require statements in there. There is also a mysterious __filename variable that is not defined anywhere in this file. Seems like some extra statements are required at the start:

import { createRequire } from 'module';
const require = createRequire(import.meta.url);

import { URL } from 'url';
const __filename = new URL('', import.meta.url).pathname;

Once I add them, the script will happily run:

$ node --experimental-vm-modules test.js
10

I couldn't find any information here about not combining PThreads + ES6 + Node, so I figured that this behavior was a bug. Bit surprised that no one's reported this before, actually; I suppose ES6 hasn't really caught on with Node users yet.

[toBeOfUse]

This is also fixable for me by renaming the *.worker.js file to *.worker.cjs and also changing the filename where it's used in the primary *.mjs file. (Edit: Actually, this breaks compatibility with the browser because the .cjs file has an application/node MIME type, which browsers don't expect...) Either the .worker.js file should become an ES6 module itself, or at least it should get a filename that reflects that it's a CommonJS module in an ES6 environment

[mmakhalaf]

Is there any progress on this issue please? I was able to work around this using @toBeOfUse as a test but it does break the browser build.

[mmomtchev]

My solution is to include a dummy package.json in the build directory which contains only:

{
  "type": "commonjs"
}

[kleisauke]

I think this has been fixed with PR #21041, available in the (upcoming) v3.1.52 release.

[sbc100]

Lets close this out for now and we can re-open if it doesn't actually fix the issue.

[guyutongxue]

@sbc100 Hi, I'm facing this issue in v3.1.52, when using -s EXPORT_ES6 -pthread with -O3. Seems that Acorn optimizer does not recognize global scope await introduced in #21041.

guyutongxue@GuyuDesktop:~$ cat temp.c
int main() {}

guyutongxue@GuyuDesktop:~$ emcc temp.c -o temp.html -O3 -pthread -s EXPORT_ES6 -s MODULARIZE
file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:3567
  var err = new SyntaxError(message);
            ^

SyntaxError: Cannot use keyword 'await' outside an async function (24:28)
  const { createRequire } = await import('module');
                            ^

    at Parser.pp$4.raise (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:3567:13)
    at Parser.pp$5.checkUnreserved (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:3465:14)
    at Parser.pp$5.parseIdent (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:3479:10)
    at Parser.pp$5.parseExprAtom (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:2849:19)
    at Parser.pp$5.parseExprSubscripts (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:2709:19)
    at Parser.pp$5.parseMaybeUnary (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:2675:17)
    at Parser.pp$5.parseExprOps (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:2602:19)
    at Parser.pp$5.parseMaybeConditional (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:2585:19)
    at Parser.pp$5.parseMaybeAssign (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:2552:19)
    at Parser.pp$8.parseVar (file:///home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/node_modules/acorn/dist/acorn.mjs:1315:24) {
  pos: 836,
  loc: Position { line: 24, column: 28 },
  raisedAt: 848
}
emcc: error: '/home/guyutongxue/clangd-wasm-build/emsdk/node/16.20.0_64bit/bin/node /home/guyutongxue/clangd-wasm-build/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /home/guyutongxue/temp.worker.mjs minifyWhitespace --exportES6' failed (returned 1)

[sbc100]

I believe that should be fixed in #21117. That change will be in the next release (3.1.53) or you can use emsdk install tot to get it today.

[guyutongxue]

I believe that should be fixed in #21117. That change will be in the next release (3.1.53) or you can use emsdk install tot to get it today.

Nice, thank you 😄