Interoperable support for UTF-8 BOM

[stof]

🤔 What's the problem you're trying to solve?

A comment in #398 (comment) brought the question of the support for UTF-8 BOM.

My quick investigation makes me think that the Java parser supports file using an UTF-8 BOM by stripping it from the parsed source (probably related to its non-standard support for other encodings than UTF-8), while other parsers don't seem to support it.

✨ What's your proposed solution?

Should all parsers add support for such BOM-stripping to support UTF-8 files encoded with a BOM ?

⛏ Have you considered any alternatives or workarounds?

No response

📚 Any additional context?

No response

[mpkorstanje]

Should all parsers add support for such BOM-stripping to support UTF-8 files encoded with a BOM.

No.

Feature files should use UTF-8 encoding without a BOM.

• https://superuser.com/questions/1553666/is-utf-8-with-bom-the-old-utf-what-is-utf-8-now

Specifically because:

The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."

Parser implementations may strip the U+FEFF. Though now that windows supports saving texts in Notepad without a BOM, I don't think that feature will be needed for new implementations.

[304NotModified]

In my opinion, the BOM should be supported but optional.

Some editors write a BOM automatically, and it’s still the most reliable indicator of a UTF file’s encoding.

Because the BOM is fairly complex and usually invisible to end users, I don’t think we should make their lives harder by rejecting files that include it.

[mpkorstanje]

Some editors write a BOM automatically

I think I that was just Notepad on Windows. Are there others?

I don’t think we should make their lives harder by rejecting files that include it.

Before we make our own lives harder than they need to be, do you have an example of a user that has to deal with BOMs?

Though given that you're in the .Net ecosystem, I'd accept a PR for .Net to strip any leadingU+FEFF either way.