doc: add WHATWG file URLs in fs module #12670

anchnk · 2017-04-26T15:59:42Z

Update fs module documentation adding WHATWG file URLs support for relevant fs functions/classes.

Fixes: #12341
Refs: #10739

Checklist

documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

doc, fs

Description of change

Add WHATWG file URLs support in fs root level.

Where relevant :

Update YAML flags in order to update revision history.
State that support for this feature is currently experimental.
Update fs functions / classes arguments types.

jasnell

Minor rewording suggestion. Thank you very much for doing this! :-)

jasnell · 2017-04-26T16:08:35Z

doc/api/fs.md

+Most fs functions' path or filename arguments can be WHATWG URLs. Only
+WHATWG URLs using `file:///` protocol are supported.
+
+*Note* that file WHATWG file URLs are always absolute paths. WHATWG file URLs


Suggestion:

For most `fs` module functions, the `path` or `filename` argument may be passed as a WHATWG [URL][] object. Only [URL][] objects using the `file:` protocol are supported. Use of [URL][] objects is currently still experimental. *Note*: `file:` URLs are always absolute paths.

Then add a link at the end of the page:

[URL]: url.html#url_the_whatwg_url_api

anchnk · 2017-04-26T16:30:08Z

@jasnell Updated the PR in order to include your rewording suggestion !

Thanks a lot for helping me out 👍

TimothyGu · 2017-04-26T17:00:10Z

doc/api/fs.md

 -->

 * `eventType` {string} The type of fs change
-* `filename` {string|Buffer} The filename that changed (if relevant/available)
+* `filename` {string|Buffer|URL} The filename that changed (if relevant/available)


This is not documentation for a method, but description of the payload of an event. Hence it cannot be a URL.

Absolutely right, on my way to correct that.

addaleax

LGTM with @TimothyGu’s comment addressed

addaleax · 2017-04-26T18:30:30Z

doc/api/fs.md

@@ -1646,7 +1776,7 @@ changes:
    description: The `cache` parameter was removed.
 -->

-* `path` {string|Buffer};
+* `path` {string|Buffer|URL};


I think the ; here is a typo, you can remove it while you’re already here if you want

Good catch @addaleax fixing it as well

anchnk · 2017-04-26T19:28:59Z

Pushed a new commit including suggestions from @TimothyGu and @addaleax
Also I think the PR has to be run in the CI linter ?
Somebody can take care of that ?
Thank you all for your hints and help and keep going if needed.

addaleax · 2017-04-26T19:32:49Z

Also I think the PR has to be run in the CI linter ?

It’s a docs-only change, I think we’re good. :) We may have linting for the docs at some point in the future, but we haven’t integrated anything into CI or make test yet.

benjamingr · 2017-04-26T22:22:24Z

Looks good but would love to see some examples added.

TimothyGu

Almost there!

TimothyGu · 2017-04-26T22:54:17Z

doc/api/fs.md

@@ -2495,6 +2687,7 @@ The following constants are meant for use with the [`fs.Stats`][] object's
 [`stat()`]: fs.html#fs_fs_stat_path_callback
 [`util.inspect(stats)`]: util.html#util_util_inspect_object_options
 [`WriteStream`]: #fs_class_fs_writestream
+[URL]: url.html#url_the_whatwg_url_api


This should maintain the alphabetical order.

Hi @TimothyGu what alphabetical order ? I was trying to figure it out if there was something like that when I included that change. However i couldn't guess what the logic was as list items seems a bit melted.

Does internal vs external links matters in the order ?
Does functions / classes matters ?

If you can describe me the logical structure beneath the list I will be happy to sort that :)

the links are un-ordered ATM, sadly, this seems as good a place as any for now

Thank you @sam-github. I will wait that your PR is merged to rebase it in mine.

TimothyGu · 2017-04-26T22:56:42Z

doc/api/fs.md

@@ -70,6 +70,12 @@ the entire process until they complete--halting all connections.
 The relative path to a filename can be used. Remember, however, that this path
 will be relative to `process.cwd()`.

+For most `fs` module functions, the `path` or `filename` argument may be passed
+as a WHATWG [URL][] object. Only [URL][] objects using the `file:` protocol are
+supported. Use of [URL][] objects is currently still experimental.


Aside from the scheme limitation, we also have some other platform-specific restrictions. Documenting them through examples can be helpful.

I second that. Examples may be helpful. Will work on that tomorrow. Thanks a lot !

link experimental to the stability index, pls

@sam-github You mean link experimental to documentation.html#documentation_stability_index ?

anchnk · 2017-04-27T13:52:15Z

Created this gist with proposal of examples. Feedback is welcome ! Also I plan to add James description in his PR #10739 below the short description i previously made at the root module level.

On Windows, file: URLs with a hostname convert to UNC paths, while file: URLs with drive letters convert to local absolute paths:
file://hostname/a/b/c => \hostname\a\b\c
file:///c:/a/b/c => c:\a\b\c
On all other platforms, file: URLs with a hostname are unsupported and will result in a throw:
file://hostname/a/b/c => throw!
file:///a/b/c => /a/b/c

sam-github · 2017-04-27T14:30:33Z

Please don't use experimental to describe this feature. That word has a precise definition in node documentation: https://nodejs.org/api/documentation.html#documentation_stability_index and this feature is not gated by a command line flag.

Fwiw, I'm not sure why we need this feature. Unlike https://www.npmjs.com/package/open-uri you need to already have parsed a url into a URL object, checked its scheme and verified its a file.... and at that point, why not just pass the file path from the url directly to the fs call?

addaleax · 2017-04-27T14:34:35Z

Please don't use experimental to describe this feature.

The WHATWG URL API is experimental, the usage of the term in this PR is correct. In this case, it’s the description of “experimental” in the Stability index that’s wrong – we haven’t been consistent about putting experimental features behind flags (which I am okay with, it can be a case-by-case decision, but we should probably document that better).

Review was for an older iteration of the PR

jasnell · 2017-04-27T15:55:27Z

The WHATWG URL stuff is still technically experimental, although I'm likely to be opening a PR very soon (ahem, later today) that proposes upgrading it to a fully supported feature.

sam-github · 2017-04-27T17:34:38Z

I think the URL stuff is technically NOT experimental :-). But as @jasnell says, it will be irrelevant soon.

@anchika That still means you should remove the experimental text.

@addaleax I'm opening an issue seeking confirmation that the text is correct as-is. Until the definition is changed, it is what it is. Full disclosure, I think that we should hide all experimental features behind a flag. Maybe we should talk about at collab summit.

#10739 is absent of any information on why the feature should be added, I still think its odd. Not sure what use-case its for. Does it help browser compatibility? But they don't have fs. Hm, :puzzled-face:.

"experimental" status or not, I doubt we are going to delete this feature now that its here, perhaps the docs can at least say what its useful for.

Also, does the arg actually have to be a URL object, or can it just be an object that has some properties "like" a URL object?

jasnell · 2017-04-27T18:40:55Z

@sam-github ... this PR should be ready to land before my PR removing the experimental status so the experimental text should actually remain in this PR.

does the arg actually have to be a URL

In practice, yes. See https://github.com/nodejs/node/blob/master/lib/internal/url.js#L1348. The code checks for the presence of the [searchParams] Symbol which is hidden within the internal/url.js file. It would be exceedingly unlikely (tho not impossible) for someone to construct an object that looks like a URL that would work with this.

anchnk · 2017-04-27T22:41:16Z

Pushed a new commit with platform specific description and examples as requested by @TimothyGu.
For now I will keep the text stating the feature is currently experimental.
Things might change about that but for now it seems ok as it.
I still have to sort the URL link at the end of the file but I can't figure what is the actual sorting order of links. Can somebody help me with that ?

sam-github · 2017-04-28T17:36:32Z

doc/api/fs.md

+  - version: v7.6.0
+    pr-url: https://github.com/nodejs/node/pull/10739
+    description: The `path` parameter can be a WHATWG `URL` object using `file:`
+                 protocol. Support is currently still *experimental*.


stability statement are usually linked to stability definitions

@sam-github Same as above link the word experimental to documentation.html#documentation_stability_index

jasnell · 2017-04-28T21:08:57Z

@anchnk ... looks like this needs a quick rebase :-)

anchnk · 2017-04-28T21:11:41Z

@jasnell yup already done locally with amazing support from @addaleax
I will push it once I have sam details so I can enhance the experimental linking :)

TimothyGu

This PR still doesn't address restrictions like these:

// On POSIX
fs.readFileSync(new URL('file:///path/%2F'));
// TypeError [ERR_INVALID_FILE_URL_PATH]: File URL path must not include encoded / characters

// On Windows
fs.readFileSync(new URL('file:///C:/path/%2F'));
fs.readFileSync(new URL('file:///C:/path/%5C'));
// TypeError [ERR_INVALID_FILE_URL_PATH]: File URL path must not include encoded \ or / characters

// On Windows
fs.readFileSync(new URL('file:///notdriveletter/path/%5C'));
// TypeError [ERR_INVALID_FILE_URL_PATH]: File URL path must be absolute

You might want to split this part of the documentation into a new chapter "WHATWG URL object support" (or something like that). The section has grown too big IMO.

TimothyGu · 2017-04-29T17:58:37Z

doc/api/fs.md

+supported. Use of [URL][] objects is currently still experimental.
+
+```js
+const fs  = require('fs');


nit: extra space before =

fixed with last commits

TimothyGu · 2017-04-29T17:58:46Z

doc/api/fs.md

+```js
+const fs  = require('fs');
+const { URL } = require('url');
+const file_url = new URL('file:///tmp/hello');


camelCase please.

fixed with last commits

TimothyGu · 2017-04-29T17:59:52Z

doc/api/fs.md

+const file_url_with_hostname = new URL('file://hostname/p/a/t/h/file');
+
+// path with drive letter converted to absolute path => c:\tmp\hello
+const file_url_with_drive_letter = new URL('file:///c:/tmp/hello');


It seems more common to have the drive letter capitalized.

TimothyGu · 2017-04-29T18:00:37Z

doc/api/fs.md

+const { URL } = require('url');
+
+// path with hostname => throw!
+const file_url_with_hostname = new URL(`file://hostname/p/a/t/h/file`);


Template string literal is unnecessary here. A ' should suffice.

TimothyGu · 2017-04-29T18:02:02Z

doc/api/fs.md

+// path with drive letter converted to absolute path => c:\tmp\hello
+const file_url_with_drive_letter = new URL('file:///c:/tmp/hello');
+
+// On windows, WHATWG file URLs with hostname convert to UNC path


TimothyGu · 2017-04-29T18:02:58Z

doc/api/fs.md

+const { URL } = require('url');
+const file_url = new URL('file:///tmp/hello');
+
+fs.readFile(file_url, 'utf-8', (err,data) => {


IMO it would be easier to use the synchronous API (fs.readFileSync) instead. You can even leave off the encoding argument since it's not necessary to demonstrate URL support.

fs.readFileSync(fileURL);

In the last commits I tried to remove everything that wasn't meaningful for the examples. So now they are quite minimalistic. That was based on your suggestion, again let me know.

TimothyGu · 2017-04-29T18:03:24Z

doc/api/fs.md

@@ -70,6 +70,76 @@ the entire process until they complete--halting all connections.
 The relative path to a filename can be used. Remember, however, that this path
 will be relative to `process.cwd()`.

+For most `fs` module functions, the `path` or `filename` argument may be passed
+as a WHATWG [URL][] object. Only [URL][] objects using the `file:` protocol are


[`URL`][] object

I created a new chapter on the last commit based on your suggestion @TimothyGu . Also the URL link formatting is fixed.

addaleax · 2017-04-29T23:54:41Z

doc/api/fs.md

+fs.readFileSync(new URL('file:///p/a/t/h/%2F'));
+fs.readFileSync(new URL('file:///p/a/t/h/%2f'));
+/* TypeError [ERR_INVALID_FILE_URL_PATH]: File URL path must not include encoded
+/ characters *:


You are missing a */ here :)

Oups ! thank you fixed !

addaleax · 2017-04-29T23:54:55Z

doc/api/fs.md

+/* TypeError [ERR_INVALID_FILE_URL_PATH]: File URL path must not include encoded
+/ characters *:
+```
+On Windows, `file:` URLs having encoded backslash will result in a throw :


Can you make sure to not have whitespace before :? ;)

anchnk · 2017-04-30T00:23:56Z

I did quite a bunch of changes in the last commits, so feel free to give any feedback.
Especially regarding the examples.
Thanks a lot for all your suggestions, help and support.

TimothyGu

One last comment. Otherwise LGTM.

TimothyGu · 2017-04-30T00:43:09Z

doc/api/fs.md

@@ -2499,6 +2774,7 @@ The following constants are meant for use with the [`fs.Stats`][] object's
 [`stat()`]: fs.html#fs_fs_stat_path_callback
 [`util.inspect(stats)`]: util.html#util_util_inspect_object_options
 [`WriteStream`]: #fs_class_fs_writestream
+[URL]: url.html#url_the_whatwg_url_api


This entry shouldn't be necessary anymore.

Fixed !
Thank you a lot.
Your feedback & suggestions greatly helped me.

TimothyGu

Other than the tiniest of nits, LGTM.

TimothyGu · 2017-04-30T15:41:00Z

doc/api/fs.md

+the drive letter. Using another separator will result in a throw.
+
+On all other platforms, `file:` URLs with a hostname are unsupported and will
+result in a throw :


Extra space before :

(This is English, not French :)

Oh so right, forgot about that difference. Sorry :) All fixed in latest commit

TimothyGu · 2017-04-30T15:41:06Z

doc/api/fs.md

+result in a throw :
+
+```js
+// On other platforms :


TimothyGu · 2017-04-30T15:41:21Z

doc/api/fs.md

+```
+
+A `file:` URL having encoded slash characters will result in a throw on all
+platforms :


TimothyGu · 2017-04-30T15:41:48Z

doc/api/fs.md

+/* TypeError [ERR_INVALID_FILE_URL_PATH]: File URL path must not include encoded
+/ characters */
+```
+On Windows, `file:` URLs having encoded backslash will result in a throw :


... and here.

jasnell · 2017-05-04T13:26:05Z

@anchnk ... can you squash the commits down to prepare for landing?

Update fs module documentation adding WHATWG file URLS support for relevant fs functions/classes. Fixes: nodejs#12341 Refs: nodejs#10739

anchnk · 2017-05-04T13:37:09Z

@jasnell Squashing done.

Update fs module documentation adding WHATWG file URLS support for relevant fs functions/classes. PR-URL: #12670 Fixes: #12341 Ref: #10739 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Timothy Gu <timothygu99@gmail.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

jasnell · 2017-05-04T13:49:55Z

Landed in c1b3b95

nodejs-github-bot added doc Issues and PRs related to the documentations. fs Issues and PRs related to the fs subsystem / file system. labels Apr 26, 2017

jasnell requested changes Apr 26, 2017

View reviewed changes

mscdex added the whatwg-url Issues and PRs related to the WHATWG URL implementation. label Apr 26, 2017

TimothyGu previously requested changes Apr 26, 2017

View reviewed changes

addaleax approved these changes Apr 26, 2017

View reviewed changes

jasnell approved these changes Apr 26, 2017

View reviewed changes

benjamingr approved these changes Apr 26, 2017

View reviewed changes

TimothyGu reviewed Apr 26, 2017

View reviewed changes

sam-github mentioned this pull request Apr 27, 2017

Confirmation that the stability definition is correct #12701

Closed

jasnell approved these changes Apr 27, 2017

View reviewed changes

sam-github reviewed Apr 28, 2017

View reviewed changes

TimothyGu suggested changes Apr 29, 2017

View reviewed changes

anchnk force-pushed the master branch from 1945531 to 2f7e4df Compare April 29, 2017 23:52

addaleax reviewed Apr 29, 2017

View reviewed changes

TimothyGu reviewed Apr 30, 2017

View reviewed changes

anchnk force-pushed the master branch from 1d2a940 to 7dd4a5f Compare April 30, 2017 10:20

TimothyGu approved these changes Apr 30, 2017

View reviewed changes

anchnk force-pushed the master branch from 7dd4a5f to 43ae938 Compare May 1, 2017 10:34

joyeecheung approved these changes May 4, 2017

View reviewed changes

doc: add WHATWG file URLs in fs module

b2257f9

Update fs module documentation adding WHATWG file URLS support for relevant fs functions/classes. Fixes: nodejs#12341 Refs: nodejs#10739

anchnk force-pushed the master branch from 43ae938 to b2257f9 Compare May 4, 2017 13:36

jasnell closed this May 4, 2017

TimothyGu added dont-land-on-v4.x labels May 8, 2017

TimothyGu mentioned this pull request May 8, 2017

file:// URLs node-fetch/node-fetch#75

Closed

jasnell mentioned this pull request May 11, 2017

8.0.0 Release Proposal #12220

Closed

kevinkassimo mentioned this pull request Apr 23, 2019

fetch() a local file denoland/deno#2150

Closed

doc: add WHATWG file URLs in fs module #12670

doc: add WHATWG file URLs in fs module #12670

Conversation

anchnk commented Apr 26, 2017 • edited Loading

Checklist

Affected core subsystem(s)

Description of change

jasnell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anchnk commented Apr 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anchnk Apr 26, 2017 • edited Loading

Choose a reason for hiding this comment

anchnk commented Apr 26, 2017

addaleax commented Apr 26, 2017

benjamingr commented Apr 26, 2017

TimothyGu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anchnk commented Apr 27, 2017

sam-github commented Apr 27, 2017

addaleax commented Apr 27, 2017

jasnell commented Apr 27, 2017

sam-github commented Apr 27, 2017

jasnell commented Apr 27, 2017

anchnk commented Apr 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented Apr 28, 2017

anchnk commented Apr 28, 2017

TimothyGu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anchnk commented Apr 30, 2017

TimothyGu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimothyGu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented May 4, 2017

anchnk commented May 4, 2017

jasnell commented May 4, 2017

anchnk commented Apr 26, 2017 •

edited

Loading

anchnk Apr 26, 2017 •

edited

Loading

TimothyGu left a comment •

edited

Loading