TextDecoder('utf-8') doesn’t match spec

[srl295]

• Version: tested v8.5.0 / master
• Subsystem: core

from discussion in #16876

in Node.js,

new (require('util').TextDecoder)('utf-8')
 .decode(Buffer.from([0xF0, 0x80, 0x80])).length === 1 // U+FFFD

But in Safari/FF/Chrome,

new TextDecoder('utf-8')
 .decode(new Uint8Array([0xF0, 0x80, 0x80])).length === 3 // U+FFFD U+FFFD U+FFFD

think Node.js is wrong here per https://encoding.spec.whatwg.org/#utf-8-decoder

//cc @mathiasbynens

[srl295]

in v8: https://bugs.chromium.org/p/v8/issues/detail?id=7055

[jungshik]

The code snippet above in web browsers(implementing TextDecoder API) should read as following:

new TextDecoder('utf-8')
 .decode(new Uint8Array([0xF0, 0x80, 0x80])).length

[srl295]

@jungshik oops… exactly right. I fixed it.

[srl295]

^ already fixed in v8 (see link above)

Yea I recently fixed this in V8. The fix is in V8 6.3.
The relevant CL is https://chromium-review.googlesource.com/671020
See also https://bugs.chromium.org/p/chromium/issues/detail?id=765608

[tniessen]

The relevant commit is v8/v8@6389b7e. Should we cherry-pick this for node 8?

[srl295]

@tniessen seems like a good idea—means the other UTF-8 handling (v8 internal, and via ICU) will be in line with TextDecoder

[TimothyGu]

This issue really has nothing to do with V8, since TextDecoder is backed by ICU. On master, this has been fixed by #16876, but not in v9.x or earlier. It seems the original issue (TextDecoder) only affects released versions of Node.js. We'll also have to make the same fix for require('string_decoder').StringDecoder, if we were to go all in.

[TimothyGu]

I guess we just need to backport

• http://bugs.icu-project.org/trac/changeset/40445/trunk/icu4c/ and
• http://bugs.icu-project.org/trac/changeset/40455/trunk/icu4c/

to v8.x, where the TextDecoder API is actually stable; or we can just upgrade ICU to 60 in v8.x LTS. We'll also need tests, as it seems WPT does not currently contain any tests for this behavior. /cc @jasnell

[srl295]

No please do not backport those commits. V8 and not icu is used for utf -8. Check the code. @jasnell

[TimothyGu]

@srl295 I have just backported the commits in my local branch, and this issue was fixed. Can you explain how V8 is used for UTF-8? If you mean

node/lib/internal/encoding.js

Line 299 in e624503

return encodeUtf8String(`${input}`);

, that is for TextEncoder, not TextDecoder.

[srl295]

Maybe we are looking at different things. The issue I saw is there even with no icu El El vie, nov. 10, 2017 a las 6:49 p. m., Timothy Gu < notifications@github.com> escribió:

…

@srl295 <https://github.com/srl295> I have just backported the commits in my local branch, and this issue was fixed. Can you explain how V8 is used for UTF-8? If you mean https://github.com/nodejs/node/blob/e6245030b193db7de9daad1e455f1fcb7ce4daac/lib/internal/encoding.js#L299, that is for Text*En*coder, not Text*De*coder. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16894 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0Msz5nNG1k08_rsokgYJWXCrFDPTWxks5s1QtVgaJpZM4QXRQa> .

[TimothyGu]

@srl295

The issue I saw is there even with no icu

no-icu uses string_decoder.StringDecoder, which as I noted in #16894 (comment) handles this wrongly as well.

[srl295]

@TimothyGu As I said please proceed how you think best if you want to take this issue.

[jasnell]

appears to be fixed in 10.x and master