Sentience tokenizer

Tiny zero-dependency tokenizer for simple DSLs and config/query languages in Rust.
Generic: drop it into parsers, rule engines, interpreters, or build tooling.
Supports identifiers, numbers, strings, operators, and a small set of keywords.
Designed for speed, clarity, and easy embedding.

Features

Zero dependencies (only std).
Token kinds: identifiers, numbers, strings, parens/brackets/braces, = + - * / ->.
Keywords: true false if then else let rule and or.
Spans included for each token.
Whitespace & // comments skipped.

Spec

Aspect	Rules
Identifiers	ASCII: `[A-Za-z_][A-Za-z0-9_]*`
Numbers	Decimal integers/decimals; optional exponent `e\|E[+\-]d+`. Single dot allowed once; `..` is not consumed by numbers.
Strings	Double-quoted. Escapes: `\n`, `\t`, `\r`, `\"`, `\\`. Unknown escapes = error.
Comments	`//` to end-of-line.
Delimiters	`(` `)` `{` `}` `[` `]` `,` `:` `;`
Operators	`=`, `+`, `-`, `*`, `/`, `->`
Keywords	`true`, `false`, `if`, `then`, `else`, `let`, `rule`, `and`, `or`

The enum TokenKind, types Token/Span, functions tokenize/tokenize_iter, LineMap, and error types LexError{Kind} are part of the stable API.

Error Reporting

Lexing errors return a LexError with kind and span. Example with LineMap:

use sentience_tokenize::{tokenize, LineMap};

let src = "\"abc\\x\"";
let map = LineMap::new(src);
let err = tokenize(src).unwrap_err();
let (line, col) = map.to_line_col(err.span.start);
println!("{}:{}: {}", line, col, err.kind.as_str());

Output

1:5: invalid escape sequence

Stable API surface

Types: TokenKind, Token, Span
Functions: tokenize(&str) -> Result<Vec<Token>, LexError>, tokenize_iter(&str)
Utilities: LineMap for byte→(line, col)
Errors: LexError, LexErrorKind

Iterator API example

use sentience_tokenize::{tokenize_iter, TokenKind};

fn main() {
    for tok in tokenize_iter("let x = 1.2e-3") {
        let t = tok.unwrap();
        if let TokenKind::Ident(name) = &t.kind {
            println!("ident: {} @{}..{}", name, t.span.start, t.span.end);
        }
    }
}

Install

Add to Cargo.toml:

[dependencies]
sentience-tokenize = "0.1"

Example

use sentience_tokenize::tokenize;

fn main() {
    let code = r#"
        // sample
        let rule greet(name) = "hi, " + name
        if true and false then x = 1 else x = 2;
    "#;

    let toks = tokenize(code).unwrap();
    for t in toks {
        println!("{:?} @{}..{}", t.kind, t.span.start, t.span.end);
    }
}

Output (truncated)

Let @18..21
Rule @22..26
Ident("greet") @27..32
LParen @32..33
Ident("name") @33..37
RParen @37..38
Eq @39..40
String("hi, ") @41..47
Plus @48..49
Ident("name") @50..54
...

Run tests

cargo test

Example binary

cargo run --example basic
cargo run --example pretty
echo 'let rule greet(n) = "hi, " + n' | cargo run --example stream

Dev

Benchmark

cargo bench

Fuzzing

Includes a cargo-fuzz setup.

cargo bench
cargo test
cargo install cargo-fuzz
cargo fuzz run tokenize -- -runs=1000

Why?

Small, standalone lexer - no macros, no regexes.
Useful as a foundation for parsers, DSLs, or interpreters.
Explicit spans for better error reporting.

sentience-tokenize 0.1.1