Syntaxmate

A Rust-native TextMate syntax highlighter with the batteries included: 264 validated languages, four accessible GitHub themes, exact scope stacks, incremental state, path detection, safe HTML/ANSI output, and no native Oniguruma dependency.

[dependencies]
syntaxmate = "0.1"

Highlight in three lines

use syntaxmate::Highlighter;

let mut highlighter = Highlighter::bundled()?;
let output = highlighter.highlight_html(
    "rust",
    "fn main() { println!(\"<hello>\"); }",
    "github-dark",
)?;
assert!(output.status().is_complete());
println!("{}", output.as_str());
# Ok::<(), syntaxmate::Error>(())

highlight_html escapes source and attributes. highlight_ansi emits 24-bit terminal colors and sanitizes source control characters by default.

Structured highlighting

Use structured spans when rendering into an editor, TUI, or custom format:

use syntaxmate::Highlighter;

let mut highlighter = Highlighter::bundled()?;
let source = "fn main() { println!(\"hello\"); }";
let document = highlighter.highlight("rust", source, "github-dark")?;

for line in document.lines() {
    for span in line.spans() {
        let scopes = line.scope_names(span.scope_stack()).collect::<Vec<_>>();
        println!("{:?} {:?} {scopes:?}", span.range(), span.style());
    }
}
# Ok::<(), syntaxmate::Error>(())

Automatic path detection and incremental sessions use the same catalog:

use syntaxmate::Highlighter;

let highlighter = Highlighter::bundled()?;
let mut session = highlighter.session("rust", "github-dark")?;
for line in ["fn main() {", "    println!(\"hello\");", "}"] {
    let output = session.highlight_line(line)?;
    assert!(output.status().is_complete());
}
# Ok::<(), syntaxmate::Error>(())

See the runnable examples, the rendering guide, and the API documentation on docs.rs.

Custom assets

Disable bundled assets when an application supplies its own grammars and themes:

syntaxmate = { version = "0.1", default-features = false }

Use GrammarRegistry, Tokenizer, and Theme::from_json for that path. Add any externally included grammars you want resolved. Missing optional includes are ignored like vscode-textmate; call GrammarRegistry::validate when you require a strict, closed include graph.

Reusing language preparation

Servers and editors that need independent tokenizers for the same language can retain immutable grammar and regex work explicitly:

use syntaxmate::{PreparedLanguage, TokenizerOptions};

let prepared = PreparedLanguage::for_bundled_language("rust")?;
let mut first = prepared.tokenizer(TokenizerOptions::default());
let mut second = prepared.tokenizer(TokenizerOptions::default());
assert_eq!(first.tokenize("let x = 1;"), second.tokenize("let x = 1;"));
# Ok::<(), syntaxmate::Error>(())

Each tokenizer has independent continuation state and bounded mutable caches. The caller-owned preparation retains the grammar closure plus static compiled patterns and candidate descriptors reached by derived tokenizers. PreparedLanguage::stats reports slot/count populations and conservative byte charges. Prepared regexes have a 64 MiB charged ceiling; candidate descriptors, scanners, and canonical injection outcomes have a combined 12 MiB ceiling and at most 1,024 descriptors. Custom registries use PreparedLanguage::new. Prefer direct Tokenizer construction for one-off work so this preparation is reclaimed with the tokenizer. Pathological custom grammar graphs that exceed the preparation-walk bounds return Error::Grammar and remain usable through the direct tokenizer API.

Feature flags

Features are additive. Release builds are offline and do not require Node. Node packages under tools/golden-oracle are development-only.

Compatibility and trust

Syntaxmate checks every public language against pinned vscode-textmate and vscode-oniguruma output. The checked-in contract currently covers 544 oracle cases across all 264 language IDs with no divergence allowlist. Public ranges are UTF-8 byte offsets; language-server semantic tokens are outside scope.

Regex and tokenizer work is bounded. Callers that require complete output must check HighlightStatus; safe fallback output is marked Degraded when a limit is exhausted. The release library performs no filesystem, network, or environment-variable access.

• Public API design
• Architecture
• Asset provenance and updates
• TextMate compatibility
• Quality contract
• Security policy
• Support policy

Syntaxmate combines Shiki-like batteries-included TextMate assets with a Syntect-like native Rust API, while using its own bounded regex engine and oracle-driven compatibility contract. See the project comparison for precise scope and non-goals.

Syntaxmate originated in the syntax engine developed for Mark. It is not affiliated with TextMate, Microsoft, Syntect, or Shiki.