Expand description
Support for tree-sitter incremental parsing, querying and highlighting of Buffers
For a given language the user needs to provide a .so file containing the compiled tree-sitter parser and a highlights .scm file for driving the highlighting.
Producing the token stream for a given buffer is handled in a multi-step process in order to support caching of tokens per-line and not baking in an explicit rendered representation (e.g. ANSI terminal escape codes) to the output.
- The file as a whole is tokenized via tree-sitter using a user provided query
- Tokens are obtained per-line using a LineIter which may be efficiently started at a non-zero line offset when needed
- The TokenIter type returned by LineIter yields RangeTokens containing the tags provided by the user in their query
- TK_DEFAULT tokens are injected between those identified by the user’s query so that the full token stream from a TokenIter will always contain the complete text of the raw buffer line
- RangeTokens are tagged byte offsets within the parent [Buffer] which may be used to extract and render sub-regions of text. In order to implement horizontal scrolling and clamping of text based on the available screen columns, a UI implementation will need to make use of unicode_width::UnicodeWidthChar in order to determine whether none, part or all of any given token should be rendered.
Structs§
- Line
Iter - Yield sub-iterators of tokens per-line in a file.
- Parser
- A dynamically loaded tree-sitter parser backed by an on disk .so file
- Range
Token - Token
Iter - An iterator of tokens for a single line.
- Tokenizer
- TsState
- Buffer level tree-sitter state for parsing and highlighting