Module syntax 

Syntax highlighting support

Producing the token stream for a given buffer is handled in a multi-step process in order to support caching of tokens per-line and not baking in an explicit rendered representation (e.g. ANSI terminal escape codes) to the output.

• The file as a whole is tokenized via tree-sitter using a user provided query
• Tokens are obtained per-line using a LineIter which may be efficiently started at a non-zero line offset when needed
• The TokenIter type returned by LineIter yields RangeTokens containing the tags provided by the user in their query
• TK_DEFAULT tokens are injected between those identified by the user’s query so that the full token stream from a TokenIter will always contain the complete text of the raw buffer line
• RangeTokens are tagged byte offsets within the parent GapBuffer which may be used to extract and render sub-regions of text. In order to implement horizontal scrolling and clamping of text based on the available screen columns, a UI implementation will need to make use of unicode_width::UnicodeWidthChar in order to determine whether none, part or all of any given token should be rendered.

Modules

re

Simple regex based highlighting by line.

ts

Support for tree-sitter incremental parsing, querying and highlighting of Buffers

Structs

LineIter

Yield sub-iterators of tokens per-line in a file.

RangeToken

SyntaxState

Buffer level state for parsing and highlighting visible lines.

TokenIter

An iterator of tokens for a single line.

Constants

TK_DEFAULT

TK_DOT

TK_EXEC

TK_LOAD