token-lang 1.0.0

Installation

[dependencies]
token-lang = "1"

Usage

A language defines its own token kind; token-lang pairs it with a Span and lets generic code classify the stream.

use token_lang::{Span, Token, TokenKind};

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Kind {
    Ident,
    Plus,
    Whitespace,
    Eof,
}

impl TokenKind for Kind {
    fn is_trivia(&self) -> bool {
        matches!(self, Kind::Whitespace)
    }
    fn is_eof(&self) -> bool {
        matches!(self, Kind::Eof)
    }
}

let tokens = [
    Token::new(Kind::Ident, Span::new(0, 1)),
    Token::new(Kind::Whitespace, Span::new(1, 2)),
    Token::new(Kind::Plus, Span::new(2, 3)),
    Token::new(Kind::Eof, Span::empty(3)),
];

// A parser skips trivia and stops at the end — without knowing the language.
let significant = tokens.iter().filter(|t| !t.is_trivia() && !t.is_eof()).count();
assert_eq!(significant, 2);

See docs/API.md for the full surface.

Status

Status: stable (1.0). v1.0.0 freezes the public surface — the Token type and the TokenKind seam — under Semantic Versioning: no breaking changes before 2.0. See the ROADMAP and docs/API.md.

Contributing

See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.