modules: Package exports validations and fallbacks #28949

guybedford · 2019-08-03T05:16:02Z

This clarifies and specifies the package "exports" proposal validations roughly in a way similar to as specified for import maps (except slightly more restrictive - no URLs or backtracking).

The validations implemented are:

Directory mappings must end in a trailing /
Backtracking below the package base is not permitted
If a user attempts to load a directory mapping below the package base, that is not permitted (pkg/map/../../below).
Targets starting with ../ or / are not permitted
Targets that are bare specifiers are not permitted
Targets that are URLs are not permitted
Targets cannot map into a node_modules path segment
Directory mappings into node_modules segments are not permitted

Instead of throwing on array targets, a very simple version of fallback arrays is added to allow for forwards compatibility in future, behaving just like import maps where the first valid match of the array is selected, based on the validation rules above.

The implementation is provided for both CJS and ESM, with associated specification changes for both as well.

Along with banning node_modules segments for exports, for consistency an associated addition to the package scope lookup algorithm is provided to avoid checking package scope package.json lookups through any node_modules path segment.

In addition a test has been included for the use of %2F in exports targets and user paths.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

guybedford · 2019-08-03T06:23:58Z

//cc @nodejs/modules-active-members

bmeck · 2019-08-03T11:55:30Z

Should we be preventing nested node_modules paths?

foo => ./node_modules/bar?

bmeck

nits / question, seems fine if we just add tests to answer the question of inner packages and avoid breakage in future. i have no strong opinions on throwing vs allowing.

lib/internal/modules/cjs/loader.js

src/module_wrap.cc

test/es-module/test-esm-exports.mjs

ljharb · 2019-08-03T15:12:42Z

doc/api/esm.md

-This is just a convention that works because `false`, just like `{}`, has no
-iterable own properties.
+
+Any invalid exports entries will be ignored. This includes exports not


Should we perhaps warn on them, instead of silently ignoring them?

The problem with warnings in Node.js is that when you warn for behaviour of packages in node_modules, there isn't much the user can do. This is in contrast to the browser, where warnings are for developers not users (console = developers), wheras in Node.js the users get the console.

So if we just spit out these warnings and we have a future-compatibility path, then it's just going to be spam, so I'd be against that.

Perhaps we can explicitly make these debug messages though under NODE_DEBUG=exports or similar in a follow-on PR.

That’s a fair point, but it’s the same with that event emitter memory leak issue.

It's not quite the same as the memory leak warning because we expect that there are invalid entries and ignoring them on older versions is exactly what should be happening on those. I would compare it to warning on unknown package.json properties. It would catch mian: lib.js but it would be much more likely to warn on all kinds of stuff that is perfectly correct.

It’s the same in that it’s not actionable to the top level app, because it originates in an installed package.

The same is true for npm audit warnings.

I think this kind of warning is important for encouraging proper ecosystem behavior.

The latest commit I just added gives the following for invalid export resolutions -

eg -

{ "exports": { "./x": "in:valid" } }

import 'pkg/x' will then provide the error:

Error: Cannot resolve package exports target 'in:valid' matched for './x' in C:\Users\Guy\Projects\node\node_modules\pkg\package.json, imported from C:\Users\Guy\Projects\node\x.mjs at Loader.resolve [as _resolve] (internal/modules/esm/default_resolve.js:59:13) at Loader.resolve (internal/modules/esm/loader.js:73:33) at Loader.getModuleJob (internal/modules/esm/loader.js:149:40) at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:43:40) at link (internal/modules/esm/module_job.js:42:36) { code: 'ERR_MODULE_NOT_FOUND'

what about for any processing of that package.json? I’d expect a warning when the package is read, not just when that path is imported.

You could imagine a package that provides one export pkg/newfeature for modern Node.js, and another pkg/legacyfeature for legacy Node.js. If we validate all exports on load we get the same issue discussed above with warnings for things that aren't used.

As mentioned, I'd be open to a top-level flag for this kind of global validation, but I'd be against making it the default behaviour for the reason that it is simply entirely unactionable to the user - you can't even post a PR to the dependency package itself like other Node.js warnings for node_modules.

again, I’m talking about invalid ones, not nonexistent ones.

I think it’s appropriate to always warn when using a package whose package.json has any exports that can’t be used in my node version. The actionable item is “stop using that package” or “upgrade node”.

Along the lines of Jan's comment above, this is like saying that an API exporting:

exports.legacy = function () { // uses fs.oldAPI(); } exports.modern = function () { // uses fs.newAPI(); }

should warn in legacy environments that the modern function exists just because it might call an undefined core library if you run pkg.modern().

There is nothing wrong with pkg.modern function existing if it isn't called in legacy environments. Similarly there should be nothing wrong with defining exports that only apply to modern use cases.

As I've mentioned, I'm happy to have a global validation mode as opt-in, but it shouldn't be the default for these reasons.

lib/internal/modules/cjs/loader.js

src/module_wrap.cc

guybedford · 2019-08-04T00:46:17Z

Re importing from node_modules, I could get behind explicitly restricting that case as well, if others agree?

bmeck · 2019-08-04T01:15:06Z

@guybedford easier to restrict for now and ease it later.

guybedford · 2019-08-04T01:39:40Z

Note that if we want to do this restriction, then we should also restrict the package boundary detection to not pass through node_modules as well. I can implement both together here.

lib/internal/modules/cjs/loader.js

ljharb · 2019-08-04T01:46:37Z

doc/api/esm.md

-This is just a convention that works because `false`, just like `{}`, has no
-iterable own properties.
+
+Any invalid exports entries will be ignored. This includes exports not


That’s a fair point, but it’s the same with that event emitter memory leak issue.

guybedford · 2019-08-04T01:58:52Z

Another interesting thing about blocking node_modules is that "exports": { "./": "./" } is no longer a noop as it opts-in to not being able to load from node_modules.

jkrems · 2019-08-04T02:24:49Z

Blocking node_modules inside the package also breaks reasonable bundles dependencies or other packages with a genuine node_modules directory inside of the package. I’m -1 on special-casing it but not strongly enough to block on it.

…

On Sat, Aug 3, 2019 at 10:00 PM Guy Bedford ***@***.***> wrote: Another interesting thing about blocking node_modules is that "exports": { "./": "./" } is no longer a noop as it opts-in to not being able to load from node_modules. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#28949?email_source=notifications&email_token=AAEKR5FOI2I7NWNWLQHRUF3QCYZ5VA5CNFSM4IJCID4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PY4ZA#issuecomment-517967460>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEKR5E6F2HFVZQWMJKMD4DQCYZ5VANCNFSM4IJCID4A> .

guybedford · 2019-08-04T03:12:49Z

Ok I've implemented the node_modules restrictions, we can always relax this later too if need be, but it is difficult to add later so this seems the cautious route.

doc/api/esm.md

lib/internal/modules/cjs/loader.js

bmeck

minor request to ensure query and hash of URL don't get included in the path check.

bmeck · 2019-08-05T13:59:03Z

@jkrems I think the problem here is we don't have an overall goal for what this feature is trying to aid. Is it trying to prevent package boundary crossing at all, is it trying to deal with package installation locations being unreliable, etc. I think the most conservative approach is the right one until we agree upon a goal so that we can relax as needed. For example, validating bundled dependencies could be checked via package.json#bundledDependencies but that expands in scope the implementation and how much of package.json node is tied to.

jkrems

Very nice!

jkrems · 2019-08-05T15:33:49Z

doc/api/modules.md

-   f. If either the key or exports[key] do not end with a slash (`/`),
-      throw "not found".
-   g. Return DIR/name/${exports[key]}${subpath.slice(key.length)}.
+   e. let RESOLVED_URL =


Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>

nodejs-github-bot · 2019-08-06T04:04:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/24854/

nodejs-github-bot · 2019-08-08T02:56:47Z

CI: https://ci.nodejs.org/job/node-test-pull-request/24887/

nodejs-github-bot · 2019-08-08T05:17:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/24888/

Trott · 2019-08-08T16:39:23Z

@ljharb LGTY?

ljharb

I still think we should be more aggressive on warning for any invalid export in any processed package.json - we can remove warnings later if needed, more easily than adding them.

guybedford · 2019-08-10T04:09:27Z

I'll aim to land this in the next day or so if no one beats me to it. Happy to continue the discussion on validations for potential follow-ups.

lib/internal/modules/cjs/loader.js

nodejs-github-bot · 2019-08-11T06:31:26Z

CI: https://ci.nodejs.org/job/node-test-pull-request/24935/

guybedford · 2019-08-12T10:31:51Z

Landed in 2103ae4.

PR-URL: #28949 Reviewed-By: Bradley Farias <bradley.meck@gmail.com> Reviewed-By: Jan Krems <jan.krems@gmail.com>

nodejs-github-bot added the c++ Issues and PRs that require attention from people who are familiar with C++. label Aug 3, 2019

guybedford requested a review from jkrems August 3, 2019 05:17

guybedford force-pushed the exports-validations branch from b158078 to 01f017b Compare August 3, 2019 06:08

bmeck suggested changes Aug 3, 2019

View reviewed changes

lib/internal/modules/cjs/loader.js Outdated Show resolved Hide resolved

src/module_wrap.cc Outdated Show resolved Hide resolved

test/es-module/test-esm-exports.mjs Show resolved Hide resolved

ljharb reviewed Aug 3, 2019

View reviewed changes

ljharb reviewed Aug 4, 2019

View reviewed changes

guybedford requested a review from bmeck August 4, 2019 03:13

vsemozhetbyt reviewed Aug 4, 2019

View reviewed changes

doc/api/esm.md Outdated Show resolved Hide resolved

vsemozhetbyt reviewed Aug 4, 2019

View reviewed changes

doc/api/esm.md Outdated Show resolved Hide resolved

bmeck reviewed Aug 5, 2019

View reviewed changes

lib/internal/modules/cjs/loader.js Outdated Show resolved Hide resolved

bmeck reviewed Aug 5, 2019

View reviewed changes

lib/internal/modules/cjs/loader.js Outdated Show resolved Hide resolved

bmeck suggested changes Aug 5, 2019

View reviewed changes

jkrems approved these changes Aug 5, 2019

View reviewed changes

guybedford and others added 7 commits August 5, 2019 21:41

module: pkg exports validations and fallbacks

fe2648c

feedback, ownproperty checks

3a1d493

node_modules filtering in exports and boundaries

7597165

linting fixes

6cdcf6c

Fixup backslash escaping

20f06d8

Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>

Closing bracket

110fa26

Co-Authored-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>

fixup double node_modules check

406cb02

guybedford force-pushed the exports-validations branch from 677ee4e to 5e58810 Compare August 6, 2019 01:50

bmeck approved these changes Aug 6, 2019

View reviewed changes

or -> and spec fixup

0274de7

ljharb approved these changes Aug 8, 2019

View reviewed changes

guybedford added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Aug 9, 2019

jkrems reviewed Aug 10, 2019

View reviewed changes

lib/internal/modules/cjs/loader.js Outdated Show resolved Hide resolved

ensure target dir mappings remain in target dir

52fb846

guybedford closed this Aug 12, 2019

guybedford deleted the exports-validations branch August 12, 2019 10:31

guybedford added a commit that referenced this pull request Aug 12, 2019

module: pkg exports validations and fallbacks

2103ae4

PR-URL: #28949 Reviewed-By: Bradley Farias <bradley.meck@gmail.com> Reviewed-By: Jan Krems <jan.krems@gmail.com>

jkrems mentioned this pull request Aug 13, 2019

Validate exports keys and values nodejs/modules#359

Closed

targos pushed a commit that referenced this pull request Aug 19, 2019

module: pkg exports validations and fallbacks

cb16229

PR-URL: #28949 Reviewed-By: Bradley Farias <bradley.meck@gmail.com> Reviewed-By: Jan Krems <jan.krems@gmail.com>

targos mentioned this pull request Aug 19, 2019

v12.9.0 release proposal #29210

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modules: Package exports validations and fallbacks #28949

modules: Package exports validations and fallbacks #28949

guybedford commented Aug 3, 2019 •

edited

Loading

guybedford commented Aug 3, 2019

bmeck commented Aug 3, 2019

bmeck left a comment

ljharb Aug 3, 2019

guybedford Aug 4, 2019

ljharb Aug 4, 2019

jkrems Aug 5, 2019

ljharb Aug 5, 2019

guybedford Aug 6, 2019

ljharb Aug 6, 2019

guybedford Aug 6, 2019

ljharb Aug 6, 2019

guybedford Aug 7, 2019 •

edited

Loading

guybedford commented Aug 4, 2019

bmeck commented Aug 4, 2019

guybedford commented Aug 4, 2019

ljharb Aug 4, 2019

guybedford commented Aug 4, 2019

jkrems commented Aug 4, 2019 via email

guybedford commented Aug 4, 2019

bmeck left a comment

bmeck commented Aug 5, 2019

jkrems left a comment

jkrems Aug 5, 2019

nodejs-github-bot commented Aug 6, 2019

nodejs-github-bot commented Aug 8, 2019

nodejs-github-bot commented Aug 8, 2019

Trott commented Aug 8, 2019

ljharb left a comment

guybedford commented Aug 10, 2019

nodejs-github-bot commented Aug 11, 2019

guybedford commented Aug 12, 2019

modules: Package exports validations and fallbacks #28949

modules: Package exports validations and fallbacks #28949

Conversation

guybedford commented Aug 3, 2019 • edited Loading

Checklist

guybedford commented Aug 3, 2019

bmeck commented Aug 3, 2019

bmeck left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guybedford Aug 7, 2019 • edited Loading

Choose a reason for hiding this comment

guybedford commented Aug 4, 2019

bmeck commented Aug 4, 2019

guybedford commented Aug 4, 2019

Choose a reason for hiding this comment

guybedford commented Aug 4, 2019

jkrems commented Aug 4, 2019 via email

guybedford commented Aug 4, 2019

bmeck left a comment

Choose a reason for hiding this comment

bmeck commented Aug 5, 2019

jkrems left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nodejs-github-bot commented Aug 6, 2019

nodejs-github-bot commented Aug 8, 2019

nodejs-github-bot commented Aug 8, 2019

Trott commented Aug 8, 2019

ljharb left a comment

Choose a reason for hiding this comment

guybedford commented Aug 10, 2019

nodejs-github-bot commented Aug 11, 2019

guybedford commented Aug 12, 2019

guybedford commented Aug 3, 2019 •

edited

Loading

guybedford Aug 7, 2019 •

edited

Loading