(Windows) FS can not handle certain characters in file name

[ZEDCWT]

Version

v20.4.0

Platform

Microsoft Windows NT 10.0.19045.0 x64

Subsystem

No response

What steps will reproduce the bug?

Run the following script

require('fs').writeFileSync('\uD83D\uDD79\uFE0F.log','Test')

Where \uD83D\uDD79\uFE0F is 🕹️

How often does it reproduce? Is there a required condition?

Always failed for v20.4.0, v20.3.1 is okay

What is the expected behavior? Why is that the expected behavior?

Run without errors

What do you see instead?

Additional information

Tried on linux, no errors

[bnoordhuis]

Libuv recently switched to WTF-8, to deal with surrogate pairs (0xD83D 0xDD79 is a surrogate pair) in Windows file paths. That change landed in v20.4.0.

cc @vtjnash

[vtjnash]

\uD83D\uDD79\uFE0F is not a legal string, since it incorrectly encodes for a surrogate, not a unicode character. The Joystick emoji encoding is:

U+1F579 ️ U+FE0F

[vtjnash]

We could potentially disable the UTF-8 transcoder error checking again though, and permit this to access the file anyways. The name would not round-trip correctly through the file-system, but that is not too significantly different from how case-sensitivity can be lost during the conversion to accessing the file system.

[ZEDCWT]

\uD83D\uDD79\uFE0F is not a legal string, since it incorrectly encodes for a surrogate, not a unicode character. The Joystick emoji encoding is:

U+1F579 ️ U+FE0F

For what reason you think it is not valid?

'\u{1F579}'==='\uD83D\uDD79' // true

This is JavaScript, they are identical.

[vtjnash]

Ah, interesting. That means node is explicitly converting the valid string in ecmascript's UCS-2 into an invalid UTF-8 string, possibly intentionally since ecmascript does not actually support this emoji and instead defines that it is composed of 2 unknown characters from the surrogate block. We could certainly change libuv to be a non-validating UTF-8 decoder so that it can still accept invalid strings as input.

https://mathiasbynens.be/notes/javascript-encoding

[rvagg]

This maybe shouldn't be closed until we get a libuv release then a Node release that includes it? We're going to collect a lot of dupes for this and they should be closed and point here until the release is done. #48813 #49042.

(Speaking from experience, I'm dealing with some failing windows CI tests since 20.4.0 https://github.com/ipld/codec-fixtures/actions/runs/5756528910/job/15606085523 for files that have ... complicated ... names).