Crate x0k_syntax 

Pure tree-sitter syntax tokenizer.

Maps source code to a flat list of semantic HighlightedToken spans (byte range + TokenKind). This crate has no rendering dependencies — it knows nothing about colors, themes, fonts, or HTML. Consumers map TokenKind to their own presentation:

• A native presenter resolves TokenKind to a theme color.
• A web presenter (such as the HTML the x0k-tangle weave emits) resolves it to a CSS class via css_class.

Tree-sitter grammars for JSON, Rust, Python, TypeScript and TSX are compiled in behind the syntax-highlight feature (default on). With the feature off, highlight always returns None and the grammar crates are not built.

The grammars are pinned to tree-sitter 0.24 (language ABI 14). A consumer that links its own tree-sitter grammars must bump in lockstep with this crate: mixing ABI versions fails at set_language.

Structs

HighlightedToken

A highlighted token with its byte range in the source.

Enums

Language

Supported languages for syntax highlighting.

TokenKind

Token types for syntax highlighting.

Functions

css_class

The stable CSS class name for a token kind, e.g. TokenKind::Keyword => "tok-keyword". This is the shared class-name contract every HTML/web presenter agrees on; the matching CSS lives in the consuming surface’s stylesheet. Native presenters ignore this and map TokenKind straight to a color.

highlight

Highlight code, returning token ranges.