neco-syntax-textmate

日本語

TextMate grammar tokenizer built on syntect: turns a line of source text into typed spans for syntax highlighting, diff coloring, or any tool that needs semantic labels on text.

How it works

Internally this crate wraps syntect and maps TextMate scope names down to a fixed 14-variant TokenKind enum. You don't deal with scope strings in your own code. Pass in a language name, feed lines one at a time, and get back byte-range spans with a kind attached.

State carries over between lines so multiline constructs (block comments, string literals, heredocs) resolve correctly. Call reset() to start fresh at the top of a file.

GrammarSet bundles the grammars. The default set includes syntect's built-in ~50 languages plus KDL, Fish, Typst, Nix, Pkl, and Mojo. You can also load additional .sublime-syntax files at runtime with load_grammar().

Usage

use neco_syntax_textmate::{GrammarSet, SyntaxHighlighter, TokenKind};

let gs = GrammarSet::default_set();
let mut hl = SyntaxHighlighter::new(&gs, "Rust").unwrap();

let spans = hl.tokenize_line("let x = 42;\n");
for span in &spans {
    if span.kind == TokenKind::Keyword {
        // "let" is here
    }
}

To detect a language from a file extension rather than naming it directly:

let lang = gs.detect_language("rs");
let mut hl = SyntaxHighlighter::new(&gs, lang.unwrap()).unwrap();

API

License

MIT