# Unicode Denormalization

published on

Alternate title: Plain-text Syntax Highlighting Using โ„ญ๐”ฌ๐”ฌ๐”ฉ ๐’ฐ๏ธ€๐“ƒ๏ธ€๐’พ๏ธ€๐’ธ๏ธ€โ„ด๏ธ€๐’น๏ธ€โ„ฏ๏ธ€ ๐•‹๐•–๐•ฉ๐•ฅ

Text-based programming languages overwhelmingly use the ASCII character set. Exceptional languages utilizing Unicode generally either a) donโ€™t provide additional capacities to input Unicode text (Python, Haskell, Swift, โ€ฆ), b) perform automatic text replacement using abbreviations (Agda, Lean, Uiua), or c) encourage some kind of dedicated keyboard / GUI setup to make Unicode input simpler (many APLs). These options all have their caveats, but broadly focus on the act making writing code more straightforward. Can the same be said for reading?

I propose the novel idea of using Unicode ASCII lookalikes in source code, and using automatic text replacement via the compiler/interpreter to enrich the plain text with additional metadata without straying far from the readability of ASCII. Is it practical? probably not. But it is fun :3

Syntax highlighting is pretty useful tool, since it allows additional semantic/syntactic metadata be made โ€œvisibleโ€ in source code, even if said source code is plain ascii. For example, rust-analyzer by default underlines variables in Rust code that are mutable. In general, this information disappears as soon as the language server is halted, though. Maybe this could be persisted to the source without compromising on readability ^_^

Possible pros: no โ€œmental translationโ€ needed to read, plain text is ubiquitous, a compiler can embed useful information that is technically redundant but quite useful.

Possible cons: font issues / inconsistent rendering, the โ€œinconsistent parseโ€ (what if a previous version of a compiler adds metadata thatโ€™s different from what a current version would add? should this be rejected, or rewritten to be in the correct style? could this create vulnerabilities much like invisible codepoints or unicode lookalikes can?)

# Candidate code points

Hereโ€™s a fairly comprehensive list of possible Unicode codepoints that you could use for this.

The Mathematical alphanumeric symbols unicode block (supplemented with letterlike symbols) provide multiple โ€œalternate fontsโ€ for the latin alphabet and digits. Note that digits are not supported in italic / calligraphic / fraktur variants.

I wouldnโ€™t recommend the sans serif / monospace variants, because the way regular ascii latin characters are rendered depends on the font, and it might overlap too strongly with these variants. The fraktur / calligraphic / blackboard bold styles are very distinctive, and I recommend using them, with some cautionโ€”they are less readable for prose.

The brace pairs โŸฆโŸงโฆƒโฆ„โฆ…โฆ† (from Miscellaneous mathematical symbols-A and Miscellaneous mathematical symbols-B) can be used in combination with the blackboard bold style. Same with โฆ‚ (U+2982) and โจŸ (U+2A1F).

Some symbols might potentially be useful as contextual ligatures of sort:

# calleth100

calleth100 is a hypothetical catlang based on this idea. (Examples of catlangs include Joy, Factor, and Forth.) Its name is a pun, readable as both โ€œcalleth 100โ€ and โ€œ๐“โ„ฏ๐“‰ h 100โ€ (where callet is cal(โ€œletโ€) aka calligraphic โ€œletโ€). The latter of course is a valid calleth100 snippet that you might find, for example, in a program working with SI units. It is pretty straightforward for a catlang, having a data stack + auxiliary stack, lexing by spaces, commenting lines out with #, and writing quotations using [ ].

The interesting bits:

Hereโ€™s an example calleth100 program in ASCII, the way youโ€™d type it:

def unless [ A bool [ A -> B ] [ A -> B ] -> B ] not if
def fact [ int -> int ]
    dup 1 <=
    [ dup 1 - fact * ]
    [ drop 1 ] unless
let minutes/hour 5 fact 2 /
def main [ -> ] minutes/hour to-string print # prints 60

And hereโ€™s what the compiler would turn it into:

๐’นโ„ฏ๐’ป ๐ฎ๐ง๐ฅ๐ž๐ฌ๐ฌ โŸฆ ๐”ธ bool โŸฆ ๐”ธ โ†’ ๐”น โŸง โŸฆ ๐”ธ โ†’ ๐”น โŸง โ†’ ๐”น โŸง not ๐ข๐Ÿ
๐’นโ„ฏ๐’ป fact โŸฆ int โ†’ int โŸง
    dup 1 <=
    [ dup 1 - fact * ]
    [ drop 1 ] ๐ฎ๐ง๐ฅ๐ž๐ฌ๐ฌ
๐“โ„ฏ๐“‰ minutes/hour 5 fact 2 /
๐’นโ„ฏ๐’ป ๐ฆ๐š๐ข๐ง โŸฆ โŸง minutes/hour to-string print # prints 60

โ€ฆso the obvious issue here is that the font makes it look kinda ugly. But it could work :) And thatโ€™s interesting!