[Pthreads] Fix worker.js in ES6 module environments #21041

kripken · 2024-01-09T00:31:31Z

This file can be an ES6 module if a package.json file indicates that all
files in the directory are.

We need to apply the same tricks as we apply to the main JS file in that
case, with createRequire etc.

Followup to #20939 and fixes the package.json discussion after that PR landed.

sbc100 · 2024-01-09T00:33:06Z

What about my idea of calling the worker.js files worker.mjs ? Does that not work for some reason?

kripken · 2024-01-09T00:37:48Z

That is a separate topic I think. It does not fix the issues here - we do actually need to fix up require etc., I verified that.

It might make sense to do for consistency, separately. But it would be a breaking change OTOH.

kripken · 2024-01-09T00:38:11Z

(I also verified it causes no harm to rename to .mjs, so if we are ok with that change, it's an option.)

RReverser · 2024-01-09T00:41:16Z

src/worker.js

@@ -32,7 +39,8 @@ if (ENVIRONMENT_IS_NODE) {
    require,
    Module,
    location: {
-      href: __filename
+      // __filename is undefined in ES6 modules
+      href: typeof __filename !== 'undefined' ? __filename : undefined


Is undefined really valid here? I think it should be import.meta.url instead.

undefined was enough to pass the test but I have no idea how this is used or what it is 😄 Thanks, I'll change it to that.

Out of curiosity, is it possible to test that?

Good question, depends if anything in our JS uses location.href to get script's own address - presumably that's why it was being polyfilled here.

I also wonder what are the situations where typeof __filename wouldn't be undefined in ES6 mode.

sbc100 · 2024-01-09T00:53:01Z

src/worker.js

@@ -18,6 +18,13 @@ var ENVIRONMENT_IS_NODE = typeof process == 'object' && typeof process.versions
 if (ENVIRONMENT_IS_NODE) {
  // Create as web-worker-like an environment as we can.

+  // See the parallel code in shell.js.
+#if EXPORT_ES6 && ENVIRONMENT_MAY_BE_WEB


I'm confused about ENVIRONMENT_MAY_BE_WEB.. surely require never works on the web does it?

Where does worker.js use other than below in the ENVIRONMENT_MAY_BE_NODE block?

It we only use require for those two lines below maybe we should replace them with dyanmic imports?

e.g.:

#if EXPORT_ES6
var fs = await import('fs');
var vm = await import('vm');
#else
var fs = require('fs');
var vm = require('vm');
#endif

Yeah in the main JS ENVIRONMENT_MAY_BE_WEB is only used because there are static imports for when we know that environment is Node.js and no other.

In this case there are no static imports to fall back to, so it should either add them or just use await import like you suggested.

Yeah, it's confusing. The comment this refers to explains more:

emscripten/src/shell.js

Lines 193 to 196 in 7a14e5b

// `require()` is no-op in an ESM module, use `createRequire()` to construct

// the require()` function. This is only necessary for multi-environment

// builds, `-sENVIRONMENT=node` emits a static import declaration instead.

// TODO: Swap all `require()`'s with `import()`'s?

My understanding of the lines is that in a pure node build we don't emit this anyhow. But in a mixed build we can't do that, and so if we build for node or web but end up running in node, then we get to this location, and need something to put here (since we couldn't put the static thing for a pure node build).

(This is all separate from this PR, of course, as it's preexisting.)

It turns out I was wrong, this is not separable. This PR was enough to fix the case of using a package.json file to force ES6 module format, but without such a file node.js uses the suffix to decide what to run. And this PR adds await import, which errors in a non-ES6 module.

To fix that the PR now emits .worker.mjs when EXPORT_ES6. That is a breaking change unfortunately, which may break all users that copy files to a production environment that use pthreads+EXPORT_ES6, but I don't see a way around it? Hopefully not many are using EXPORT_ES6, and probably few are because of issues like this, actually...

Thoughts?

I think if you use EXPORT_ES6 when it makes sense to create worker.mjs. I don't think think its a huge deal that we change this now. Maybe just mention it in the changelog.

Agreed, yeah, already added to the changelog.

If this breaking change sgty then let me know if you have any other feedback on the PR, otherwise I think it is done.

kripken · 2024-01-09T23:39:22Z

Added some trivial test updates, but also non-trivially a fix that we don't need to check for multiple environments here. That was only needed in the shell.js case because it interacts with MODULARIZE. Here things are simpler: if ES6 then in the node code we can just use await import etc., which simplifies things.

src/worker.js

Context: emscripten-core/emscripten#21041

* [wasm] Bump emscripten to 3.1.56 * Replace Module.asm with Module.wasmExports Module.asm was removed, use wasmExports instead. Context: emscripten-core/emscripten#19816 * Updates for .native.worker.js -> mjs rename Context: emscripten-core/emscripten#21041 * Update deps * Add general testing feed * Update mode deps * Update path * Use current python packages for now, we don't have newer ones The current names 3.1.34 are new enough * Keep using llvm 16 for runtime and aot compiler * Add -Wno-pre-c11-compat only for browser * Temporarily disable version checks to get further * Temporarily disable version checks to get further #2 * Disable -Wunused-command-line-argument * Update emsdk deps * Update icu dependency * Revert "Temporarily disable version checks to get further #2" This reverts commit 3f8834f. * Revert "Temporarily disable version checks to get further" This reverts commit fe1e5c6. * Fix emsdk check We use system python on osx too * Workaround wasm-opt crash * Workaround wasm-opt crash * Workaround wasm-opt crash * Fix WBT test * Feedback * Update ICU dependency * Update emscripten deps * Revert "Workaround wasm-opt crash" This reverts commit 200cf3b. * Revert "Workaround wasm-opt crash" This reverts commit 4530edf. * Revert "Workaround wasm-opt crash" This reverts commit 3593c41. * Increase tests timeout * Show test progress * Increase MT library tests timeout * Disable WBT tests with SkiaSharp * Increase helix tests timeout on browser * Increase WBT timeout * Increase initial heap sizes * Fix mono_wasm_load_runtime cwrap signature Fixes: `Uncaught ExitStatus: Assertion failed: stringToUTF8Array expects a string (got number)` * Enable XunitShowProgress for threading tasks tests * Try to reduce number of parallel AOT compilations To check whether it will improve memory issues on CI * Use new docker image for helix/windows tests * Revert "Try to reduce number of parallel AOT compilations" This reverts commit 5d9a6d2. * Reduce the timeouts * Reduce intitial heap size * use active issues for MT * Remove testing channel from nuget config, update deps * Update emsdk and icu dependencies --------- Co-authored-by: Larry Ewing <lewing@microsoft.com> Co-authored-by: pavelsavara <pavel.savara@gmail.com>

fix

86a28fe

kripken requested review from sbc100 and RReverser January 9, 2024 00:31

kripken mentioned this pull request Jan 9, 2024

Add test for ES6 + pthread + node #20939

Merged

RReverser reviewed Jan 9, 2024

View reviewed changes

sbc100 reviewed Jan 9, 2024

View reviewed changes

kripken added 4 commits January 8, 2024 16:59

feedback: import.meta.url

dc7c00f

python lint

5435bf4

fix

42cf278

use .worker.mjs when EXPORT_ES6

578e3fa

sbc100 approved these changes Jan 9, 2024

View reviewed changes

fix tests

49ad508

sbc100 approved these changes Jan 9, 2024

View reviewed changes

RReverser reviewed Jan 10, 2024

View reviewed changes

src/worker.js Outdated Show resolved Hide resolved

fix

48dc5ef

kripken merged commit 49ab23a into emscripten-core:main Jan 10, 2024
27 checks passed

kripken deleted the es6_worker_js branch January 10, 2024 20:16

kleisauke mentioned this pull request Jan 17, 2024

Emscripten fails on Node.js with PThreads and ES6 exports #18626

Closed

radekdoulik added a commit to radekdoulik/runtime that referenced this pull request Apr 5, 2024

Updates for .native.worker.js -> mjs rename

d23f6fa

Context: emscripten-core/emscripten#21041

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Pthreads] Fix worker.js in ES6 module environments #21041

[Pthreads] Fix worker.js in ES6 module environments #21041

kripken commented Jan 9, 2024 •

edited

Loading

sbc100 commented Jan 9, 2024

kripken commented Jan 9, 2024

kripken commented Jan 9, 2024

RReverser Jan 9, 2024

kripken Jan 9, 2024

RReverser Jan 10, 2024

RReverser Jan 10, 2024

sbc100 Jan 9, 2024

RReverser Jan 9, 2024

kripken Jan 9, 2024

kripken Jan 9, 2024

sbc100 Jan 9, 2024

kripken Jan 9, 2024

kripken commented Jan 9, 2024

	// `require()` is no-op in an ESM module, use `createRequire()` to construct
	// the require()` function. This is only necessary for multi-environment
	// builds, `-sENVIRONMENT=node` emits a static import declaration instead.
	// TODO: Swap all `require()`'s with `import()`'s?

[Pthreads] Fix worker.js in ES6 module environments #21041

[Pthreads] Fix worker.js in ES6 module environments #21041

Conversation

kripken commented Jan 9, 2024 • edited Loading

sbc100 commented Jan 9, 2024

kripken commented Jan 9, 2024

kripken commented Jan 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kripken commented Jan 9, 2024

kripken commented Jan 9, 2024 •

edited

Loading