[feature] source encoding

[Nikokrock]

Summary

The idea would be to support only UTF-8 for source encoding, as some other modern languages do, and also restrict the set of allowed characters for identifiers.

Optionally, making the language case-sensitive would be great.

Motivation

The current situation, which relies on switches, can make complete libraries unusable because users use a distinct encoding for their own sources. Both Rust and Go support only UTF-8 for source files.

Restricting the list of possible identifiers would ensure readability and avoid attacks based on confusion. I think the Go approach is probably the sanest: only Unicode letters, digits, and _.

Regarding identifier normalization, the compiler is supposed to use NFKC, but I think it's not fully implemented yet. Switching to case-sensitive identifiers could make the language closer to Go and Rust, and use just NFC (which is simpler, especially if the GoLang set is used for identifiers)

[sttaft]

Robin Leroy (egg.robin.leroy@gmail.com) has become a regular observer at the ARG meetings. He works for the Unicode Consortium, and is quite involved in establishing recommendations for how programming languages should use Unicode. In particular, he mentioned Unicode Standard Annex #‌31 Unicode Identifier{s} and [Pattern ]Syntax and Unicode Technical Standard #‌55 Unicode Source Code Handling. It might be worth looking at those documents, and possibly others that address the intersection between Unicode and programming languages.

[Nikokrock]

Rust is following indeed Unicode Standard Annex #‌31 (forgot to mention that in my first message). GoLang is a bit more restrictive than that

[Nikokrock]

And @sttaft thanks for the reference to: Unicode Source Code Handling (know only about the one on identifiers)

[sttaft]

Here are the direct links to these Unicode documents:

Unicode® Standard Annex # 31 -- Unicode Identifiers and Syntax [UAX31]:
https://www.unicode.org/reports/tr31/

Unicode® Technical Standard # 55 -- Unicode Source Code Handling [UTS55]:
https://www.unicode.org/reports/tr55/