Module semantic 

LSP semantic-token mapping for the lexical highlighter.

The crate::highlight pass produces HighlightKinds with byte ranges. Editors that speak the Language Server Protocol want semantic tokens: each significant token tagged with a standard token type (keyword, string, number, …) and a bitset of modifiers, then delta-encoded as (deltaLine, deltaStartChar, length, tokenType, modifiers) quintuples.

This module is transport-free and depends on nothing but the lexer/syntax crates — the sql-dialect-fmt-lsp binary maps these onto lsp_types, but the mapping itself (and its tests) live here so the highlighter and the editor adapter can never disagree on what a kind means.

It also understands Injections: a $$ … $$ body may embed JavaScript, Python, Java, Scala, or SQL (per a LANGUAGE clause). Tokens that fall inside an injection region are tagged so an editor can either re-highlight them with the embedded grammar or shade the whole body.

Structs

Injection

An embedded-language region inside a $$ … $$ body. Built by detect_injections; consumers can re-highlight the range with the embedded grammar named by language. #[non_exhaustive] so fields can be added compatibly.

LineToken

A semantic token as the LSP wire format wants it: zero-based line / UTF-16 char position with a UTF-16 length, plus the legend indices. These are absolute (not yet delta-encoded); call delta_encode for the on-wire (deltaLine, deltaStartChar, …) form. #[non_exhaustive] so fields can be added without breaking downstream matches.

ResolvedToken

A semantic token resolved to absolute byte coordinates (pre delta-encoding). #[non_exhaustive] so fields can be added without breaking downstream matches.

SemanticTokenModifiers

LSP semantic token modifiers, as a bitset. Only the few the lexical layer can justify are modelled; bits() produces the tokenModifiers value an editor decodes against its legend.

SemanticTokens

The full semantic-token result: the per-token tagging plus the embedded-language regions. #[non_exhaustive] so fields can be added without breaking downstream matches.

Enums

InjectedLanguage

A language that can be embedded in a Snowflake $$ … $$ UDF / procedure body.

SemanticTokenType

The standard LSP semantic token types this highlighter emits.

Functions

delta_encode

Delta-encode absolute LineTokens into the LSP wire form: each quintuple is (deltaLine, deltaStartChar, length, tokenType, tokenModifiers), where deltas are relative to the previous token (and deltaStartChar resets to the absolute column when the line advances). Tokens must be sorted by (line, start_char); line_tokens already produces them that way.

detect_injections

Scan input for embedded $$ … $$ bodies and tag each with its language. Routine bodies use LANGUAGE <name> ... AS $$...$$; dynamic SQL uses EXECUTE IMMEDIATE $$...$$. Bodies after AS $$...$$ with no LANGUAGE clause default to InjectedLanguage::Sql. Runs off the lexical highlighter, so it never parses or panics.

line_tokens

Lower resolved tokens to LSP LineTokens: split multi-line tokens (block comments, dollar-quoted bodies) into one token per line — the LSP encoding forbids a token spanning a newline — and compute UTF-16 positions and lengths. Output is sorted by (line, start_char).

line_tokens_utf8

Like line_tokens, but columns and token lengths are measured in UTF-8 bytes.

resolve_tokens

Resolve input to absolute-coordinate semantic tokens (no delta-encoding). Trivia, punctuation and lex errors drop out; everything else becomes a ResolvedToken tagged per semantic_token. Injections do not change a token’s type — the lexical kind already wins — but they are returned alongside by semantic_tokens so an editor can layer an embedded grammar over the body’s range.

semantic_token

The LSP token type + modifiers for a highlight kind, or None for kinds that carry no semantic token: whitespace (insignificant), punctuation (delimiters editors theme structurally), and lex errors (surfaced as diagnostics, not tokens).

semantic_tokens

Resolve input to semantic tokens and its embedded-language injection regions in one pass.

semantic_tokens_lsp

One-shot: resolve, lower to lines, and delta-encode. The vector an LSP server hands back for textDocument/semanticTokens/full (modulo the server’s wrapper type).

semantic_tokens_lsp_utf8

One-shot semantic tokens using UTF-8 columns/lengths.