pub struct Token<K> {
pub span: Span,
pub kind: K,
}Expand description
A single lexical token: a kind paired with the Span of
source it covers.
Token<K> is the shared seam between a lexer and a parser. The lexer produces
a stream of Token<K>; the parser consumes it. Neither needs to know the
other’s internals — they agree only on this pair. The kind K is the
language’s own classification (an enum of keywords, punctuation, literals,
and so on); token-lang stays language-agnostic by leaving K to the language
and owning only the pairing with a span.
A token is a classified span: K says what was lexed, the Span says
where. The type is Copy whenever K is, so a token whose kind is a plain
enum (a discriminant plus an eight-byte span) is cheap to pass by value.
Token<K> mirrors Spanned<K> but carries token
semantics: it orders by span first, so a slice of tokens sorts into source
order; its Display reads as kind @ start..end; and when K: TokenKind it
forwards the classification queries — is_trivia,
is_eof, and symbol — to the kind. Convert
between the two with the From impls when a layer wants the plain pair.
§Examples
use token_lang::{Span, Token, TokenKind};
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
enum Kind {
Ident,
Plus,
Eof,
}
impl TokenKind for Kind {
fn is_eof(&self) -> bool {
matches!(self, Kind::Eof)
}
}
// `a + b` lexed into tokens, then the end marker.
let a = Token::new(Kind::Ident, Span::new(0, 1));
let plus = Token::new(Kind::Plus, Span::new(2, 3));
let eof = Token::new(Kind::Eof, Span::new(5, 5));
assert_eq!(*a.kind(), Kind::Ident);
assert_eq!(plus.span(), Span::new(2, 3));
assert!(eof.is_eof());
// Tokens are `Copy` and sort into source order.
let mut stream = [eof, a, plus];
stream.sort();
assert_eq!(stream, [a, plus, eof]);Fields§
§span: SpanThe half-open source range this token covers.
Declared before kind so the derived ordering compares the
span first, sorting a slice of tokens into source order.
kind: KThe language-specific kind of this token.
Implementations§
Source§impl<K> Token<K>
impl<K> Token<K>
Sourcepub const fn new(kind: K, span: Span) -> Self
pub const fn new(kind: K, span: Span) -> Self
Pairs a kind with the span it was lexed from.
const, so a token can initialise a const or static table — useful for
the fixed marker tokens (an end-of-input, a synthetic delimiter) a lexer
hands out.
§Examples
use token_lang::{Span, Token};
let tok = Token::new("ident", Span::new(0, 5));
assert_eq!(*tok.kind(), "ident");
assert_eq!(tok.span(), Span::new(0, 5));Sourcepub const fn kind(&self) -> &K
pub const fn kind(&self) -> &K
Borrows this token’s kind.
§Examples
use token_lang::{Span, Token};
let tok = Token::new(42u8, Span::new(0, 1));
assert_eq!(*tok.kind(), 42);Sourcepub const fn span(&self) -> Span
pub const fn span(&self) -> Span
Returns this token’s span.
Span is Copy, so this hands back the range by value rather than
borrowing it.
§Examples
use token_lang::{Span, Token};
let tok = Token::new((), Span::new(3, 7));
assert_eq!(tok.span(), Span::new(3, 7));Sourcepub fn into_kind(self) -> K
pub fn into_kind(self) -> K
Consumes the token, returning just its kind and dropping the span.
§Examples
use token_lang::{Span, Token};
let tok = Token::new(String::from("if"), Span::new(0, 2));
assert_eq!(tok.into_kind(), "if");Sourcepub fn map<U>(self, f: impl FnOnce(K) -> U) -> Token<U>
pub fn map<U>(self, f: impl FnOnce(K) -> U) -> Token<U>
Transforms the kind with f, keeping the span unchanged.
This is how a layer lifts a token from one kind to another without losing where it came from — for example, mapping a raw lexer kind onto a coarser parser kind, or wrapping the kind in a richer type.
§Examples
use token_lang::{Span, Token};
let raw = Token::new("123", Span::new(4, 7));
let parsed = raw.map(|s| s.parse::<u32>().unwrap());
assert_eq!(*parsed.kind(), 123);
assert_eq!(parsed.span(), Span::new(4, 7));Sourcepub fn as_ref(&self) -> Token<&K>
pub fn as_ref(&self) -> Token<&K>
Borrows the kind, yielding a Token<&K> with the same span.
Mirrors Option::as_ref: it lets you inspect or map the
kind without consuming the original token.
§Examples
use token_lang::{Span, Token};
let owned = Token::new(String::from("name"), Span::new(0, 4));
let len = owned.as_ref().map(|s| s.len());
assert_eq!(*len.kind(), 4);
// `owned` is still usable.
assert_eq!(owned.kind(), "name");Source§impl<K: TokenKind> Token<K>
impl<K: TokenKind> Token<K>
Sourcepub fn is_trivia(&self) -> bool
pub fn is_trivia(&self) -> bool
Whether this token’s kind is trivia. Forwards to
TokenKind::is_trivia.
§Examples
use token_lang::{Span, Token, TokenKind};
#[derive(Clone, Copy)]
enum Kind {
Word,
Space,
}
impl TokenKind for Kind {
fn is_trivia(&self) -> bool {
matches!(self, Kind::Space)
}
}
assert!(Token::new(Kind::Space, Span::new(0, 1)).is_trivia());
assert!(!Token::new(Kind::Word, Span::new(1, 5)).is_trivia());Sourcepub fn is_eof(&self) -> bool
pub fn is_eof(&self) -> bool
Whether this token’s kind is the end-of-input marker. Forwards to
TokenKind::is_eof.
§Examples
use token_lang::{Span, Token, TokenKind};
#[derive(Clone, Copy)]
enum Kind {
Word,
Eof,
}
impl TokenKind for Kind {
fn is_eof(&self) -> bool {
matches!(self, Kind::Eof)
}
}
assert!(Token::new(Kind::Eof, Span::empty(9)).is_eof());
assert!(!Token::new(Kind::Word, Span::new(0, 4)).is_eof());Sourcepub fn symbol(&self) -> Option<Symbol>
pub fn symbol(&self) -> Option<Symbol>
The interned lexeme this token carries, if any. Forwards to
TokenKind::symbol.
§Examples
use intern_lang::Interner;
use token_lang::{Span, Symbol, Token, TokenKind};
#[derive(Clone, Copy)]
enum Kind {
Ident(Symbol),
Plus,
}
impl TokenKind for Kind {
fn symbol(&self) -> Option<Symbol> {
match self {
Kind::Ident(s) => Some(*s),
Kind::Plus => None,
}
}
}
let mut interner = Interner::new();
let tok = Token::new(Kind::Ident(interner.intern("x")), Span::new(0, 1));
assert_eq!(tok.symbol().and_then(|s| interner.resolve(s)), Some("x"));
assert_eq!(Token::new(Kind::Plus, Span::new(1, 2)).symbol(), None);