Module syntax 

Syntax highlighting for a fenced code block: the lines of a block and the language its fence names in, a Token per byte range out.

The grammars are Sublime Text’s, run by syntect, and the set is two-face’s — bat’s 213, which is the set plates publishes with, so a block that highlights in leaf highlights on the published page and the other way round. What comes out is not a colour and not a scope: it is the eight-way Token classing, which is all a palette can usefully tell apart, decided here once so that four frontends do not each read TextMate scope names.

How a scope becomes a token

syntect’s parser leaves a stack of scopes over every byte — source.rust meta.function.rust string.quoted.double.rust punctuation.definition.string.begin.rust over the " that opens a string literal in a function body. The token for a byte is decided the way a stylesheet over classed HTML decides it, because that is what plates’ stylesheet is and the two should agree:

• Innermost scope first. The nearest scope that names a token wins, as an inner <span>’s own colour beats what it inherits from an outer one.
• Any atom, not the first. punctuation.definition.string.begin has both punctuation and string among its atoms, and the later of the two in Token::ALL wins — so the quote reads as string, and a // reads as comment.
• meta and source name nothing. meta.function wraps a whole function, signature and body together; colouring it would flood everything nested inside. They are simply absent from the vocabulary.

What it costs

The syntax set is unpacked from its embedded dump on first use — tens of milliseconds and about a megabyte of memory — and then shared for the life of the process. Parsing itself is per block, once per rebuild of that block: the WYSIWYG map’s block cache keeps a block’s rows across edits elsewhere in the document, so typing in a paragraph never re-highlights the code above it, and typing in a code block re-highlights that block alone.

Functions

highlight

Highlight the lines of one fenced code block written in lang.

knows

Whether lang — a fence’s info string, trimmed — names a grammar: rust, rs, zig and swift do; text, "" and no-such-language do not.

Type Aliases

LineTokens

One line’s highlighting: byte ranges into that line, ascending and non-overlapping, with the token over each. A byte no range covers carries no token.