If you want to try it for yourself, here (you’ll need Node):
1. Fire up Windows Terminal
2. echo 'incompetence' > perfected
3. 'fs=require("fs"); a=fs.readFileSync("perfected", "utf-8"); console.log(a)' | node
PS. In case anyone wants the workaround:
echo 'incompetence' | Out-File -Encoding ASCII perfected
@aral is that the unicode bom
@aral does this get better if you use conemu?
(I basically cannot use the win10 vm thing that I have for work, or any win10, without conemu.)
You can install via chocolately... or of course via something else but I tend to use chocolatey because it's at least SOMEWHAT similar to a more sensible modern install system. SOMEWHAT. And yes all install systems suck...
Anyway am wondering if it is the abysmal thing that is the standard cmd.exe window, or if it is better with another...
oh actually you could see if it works in powershell first, that uses a different mechanism for doing console bits. It even supports truecolor. It still sucks (:
@mjd @aral judging by the comments on https://stackoverflow.com/questions/48848841/how-do-i-write-utf8-with-no-bom-to-console-no-file/48859679#48859679 even if you change the encoding for redirection in #PowerShell, it will still include the #ByteOrderMark. Though it might be possible in #PowerShellCore.
Doesn't take away though that these are horrible defaults and should just work regardless of what terminal you use...
I wonder if you can force it to use utf8 for files, instead of "oh, you want to use utf8! I'll make that utf16 for you."
will work, but that is just... silly.
65000 is UTF7.
I'd just go for "don't use Windows" (:
And for testing under Windows, ssh from your POP!os machine to the windows machine, and run it there... and see if THAT also has the BOM.
Ugh. I'll bet we run into some goofy things with the new product because of this UTF16 garbage and the Windows program that I wish we didn't have to use... but not my choice so...
@aral Interesting. Probably a UTF-8 BOM, which is a stupid idea by people who don't understand Unicode, namely Microsoft.
@aral Except that reading Wikipedia page further I realise that it probably isn't even BOM in UTF-8, but in UTF-16. Which is not surprising since UTF-16 is the encoding Windows uses internally. But that doesn't mean that node is broken either. You are just trying to read UTF-16 as UTF-8.
@aral that's probably only in Powershell, right?
Doesn't seem to do that in cmd - I guess it's a powershell bug?
@dheadshot Could be :)
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!