unable to deal with file names that have invalid encoding

[andrewrk]

There is a file in my user's file system and io.js cannot use any of the fs API on it, because the file name has an invalid encoding. File names are byte arrays; not strings. This is the root of the problem.

andrewrk/groovebasin#383

How to reproduce (use linux):

$ wget https://s3.amazonaws.com/superjoe/temp/encoding-test.tar.gz
$ tar xvf encoding-test.tar.gz 
encoding-test/
encoding-test/test.js
encoding-test/dir/
encoding-test/dir/\377
$ cd encoding-test/
$ cat test.js
var fs = require('fs');
var path = require('path');
var dir = 'dir/';
var list = fs.readdirSync(dir);
var filename = path.join(dir, list[0]);
console.log(filename);
console.log(fs.lstatSync(filename));
$ node test.js 
dir/�

fs.js:688
  return binding.lstat(pathModule._makeLong(path));
                 ^
Error: ENOENT, no such file or directory 'dir/�'
    at Object.fs.lstatSync (fs.js:688:18)
    at Object.<anonymous> (/home/andy/Downloads/encoding-test/test.js:7:16)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:906:3

[bnoordhuis]

It's a known issue, see e.g. nodejs/node-v0.x-archive#2387. (There is another, older issue but I can't find it just now.)

There is currently no workaround. Fixing it probably entails rewriting src/node_file.cc and lib/fs.js to accept both buffers and strings (and, perhaps, typed arrays.)

Most platforms treat file paths as byte arrays but Windows and OS X do not; I'm not sure if that would affect node.js but it's something to keep in mind. It's possible that the Windows port of libuv already takes care of all the details but the OS X port most assuredly does not.

At any rate, pull requests are welcome!

[piscisaureus]

Libuv for windows assumes that filenames are valid UTF-16 and converts those to and from UTF-8.
However on windows are really uint16 arrays; they are not necessarily valid UTF-16, so a similar issue exists on windows.

[chrisdickinson]

If possible I'd like to avoid duplicating issues from the node tracker into the iojs tracker. Closing this in favor of the original issue.

[andrewrk]

I don't understand, isn't io.js a fork? Shouldn't it have independent issues? Let's leave the issue open in io.js until it's resolved. I'll see about making a pull request to make src/node_file.cc and lib/fs.js accept buffers in addition to strings.