# Unicode Denormalization
- published on
Alternate title: Plain-text Syntax Highlighting Using
Text-based programming languages overwhelmingly use the ASCII character set. Exceptional languages utilizing Unicode generally either a) donโt provide additional capacities to input Unicode text (Python, Haskell, Swift, โฆ), b) perform automatic text replacement using abbreviations (Agda, Lean, Uiua), or c) encourage some kind of dedicated keyboard / GUI setup to make Unicode input simpler (many APLs). These options all have their caveats, but broadly focus on the act making writing code more straightforward. Can the same be said for reading?
I propose the novel idea of using Unicode ASCII lookalikes in source code, and using automatic text replacement via the compiler/interpreter to enrich the plain text with additional metadata without straying far from the readability of ASCII. Is it practical? probably not. But it is fun :3
Syntax highlighting is pretty useful tool, since it allows additional semantic/syntactic metadata be made โvisibleโ in source code, even if said source code is plain ascii. For example, rust-analyzer by default underlines variables in Rust code that are mutable. In general, this information disappears as soon as the language server is halted, though. Maybe this could be persisted to the source without compromising on readability ^_^
Possible pros: no โmental translationโ needed to read, plain text is ubiquitous, a compiler can embed useful information that is technically redundant but quite useful.
Possible cons: font issues / inconsistent rendering, the โinconsistent parseโ (what if a previous version of a compiler adds metadata thatโs different from what a current version would add? should this be rejected, or rewritten to be in the correct style? could this create vulnerabilities much like invisible codepoints or unicode lookalikes can?)
# Candidate code points
Hereโs a fairly comprehensive list of possible Unicode codepoints that you could use for this.
The Mathematical alphanumeric symbols unicode block (supplemented with letterlike symbols) provide multiple โalternate fontsโ for the latin alphabet and digits. Note that digits are not supported in italic / calligraphic / fraktur variants.
- Bold:
- Italic:
- Bold italic:
- Calligraphic:
- Bold calligraphic:
- Fraktur:
- Bold fraktur:
- Blackboard bold:
- Sans serif:
- Bold sans serif:
- Italic sans serif:
- Bold italic sans serif:
- Monospace:
I wouldnโt recommend the sans serif / monospace variants, because the way regular ascii latin characters are rendered depends on the font, and it might overlap too strongly with these variants. The fraktur / calligraphic / blackboard bold styles are very distinctive, and I recommend using them, with some cautionโthey are less readable for prose.
The brace pairs (from Miscellaneous mathematical symbols-A and Miscellaneous mathematical symbols-B) can be used in combination with the blackboard bold style. Same with โฆ (U+2982) and โจ (U+2A1F).
Some symbols might potentially be useful as contextual ligatures of sort:
- (U+2237) for
:: - (U+2254) for
:= - (U+2255) for
=: - (U+2A74) for
::= - (U+2239) for
-: - โฉต (U+2A75) for
== - โฉถ (U+2A76) for
=== - (U+203C) for
!! - (U+2049) for
!? - (U+2047) for
?? - (U+2048) for
?! - โฌ (U+226C) for
()or perhaps)( - โชค (U+2AA5) or โชฅ (U+2AA5) for
>< - โฆ and โฆ (U+2993 and U+2994) for
<(and)> - โฆ and โฆ (U+2995 and U+2996) for
>((and))<(interesting potential bracket pairs)
# calleth100
calleth100 is a hypothetical catlang based on this idea. (Examples of catlangs include Joy, Factor, and Forth.) Its name is a pun, readable as both โcalleth 100โ and โ๐โฏ๐ h 100โ (where callet is cal(โletโ) aka calligraphic โletโ). The latter of course is a valid calleth100 snippet that you might find, for example, in a program working with SI units. It is pretty straightforward for a catlang, having a data stack + auxiliary stack, lexing by spaces, commenting lines out with #, and writing quotations using [ ].
The interesting bits:
- Keywords are formatted to calligraphic script:
๐นโฏ๐ปfor general word definitions and๐โฏ๐for constant definitions. - Words are formatted to bold if they โescapeโ. Escaping is defined as either applying an escaping word to a quotation argument, or having an uneven stack effect on the auxiliary stack. The following builtin words escape:
๐๐ฉ๐ฉ๐ฅ๐ฒ,๐ข๐,๐ฉ๐ฎ๐ฌ๐ก(data to aux),๐ฉ๐จ๐ฉ(aux to data). Themaindefinition is also always bold.- This is a nice indicator for functions that mess with control flow in some way. Itโs also nicely encapsulated in the sense that escaping only propagates up word definitions if itโs not closed off tidily.
- It has types and type annotations.
- They also use square brackets (as do all function types) but they are formatted to be .
- Function types use
->, which formats to theโsymbol. - Type variables, as well as stack variables (for stack polymorphism) are formatted to .
Hereโs an example calleth100 program in ASCII, the way youโd type it:
def unless [ A bool [ A -> B ] [ A -> B ] -> B ] not if
def fact [ int -> int ]
dup 1 <=
[ dup 1 - fact * ]
[ drop 1 ] unless
let minutes/hour 5 fact 2 /
def main [ -> ] minutes/hour to-string print # prints 60And hereโs what the compiler would turn it into:
๐นโฏ๐ป ๐ฎ๐ง๐ฅ๐๐ฌ๐ฌ โฆ ๐ธ bool โฆ ๐ธ โ ๐น โง โฆ ๐ธ โ ๐น โง โ ๐น โง not ๐ข๐
๐นโฏ๐ป fact โฆ int โ int โง
dup 1 <=
[ dup 1 - fact * ]
[ drop 1 ] ๐ฎ๐ง๐ฅ๐๐ฌ๐ฌ
๐โฏ๐ minutes/hour 5 fact 2 /
๐นโฏ๐ป ๐ฆ๐๐ข๐ง โฆ โง minutes/hour to-string print # prints 60โฆso the obvious issue here is that the font makes it look kinda ugly. But it could work :) And thatโs interesting!