Struct Token 

pub struct Token<'a> {
    pub text: &'a str,
    pub span: Range<usize>,
    pub char_span: Range<usize>,
    pub kind: TokenKind,
}

A single token produced by crate::Tokenizer::segment.

The text field is a zero-copy slice of the original input string. Two span types are provided: span for byte offsets (suitable for slicing &str) and char_span for Unicode scalar-value offsets (suitable for Python/JavaScript string indexing and display).

Example

use kham_core::Tokenizer;

let tok = Tokenizer::new();
let input = "ธนาคาร100แห่ง";
let tokens = tok.segment(input);
for t in &tokens {
    // byte span slices the original string exactly
    assert_eq!(&input[t.span.clone()], t.text);
    // char span equals the Unicode scalar-value count
    assert_eq!(t.char_span.end - t.char_span.start, t.text.chars().count());
}

Fields

text: &'a str

Zero-copy reference into the original input.

span: Range<usize>

Byte offsets start..end in the original input string. Both boundaries are valid UTF-8 code-point boundaries.

char_span: Range<usize>

Unicode scalar-value (char) offsets start..end in the original input. Use these for language-level string indexing in Python, JavaScript, etc.

kind: TokenKind

Script / category of this token.

Implementations

impl<'a> Token<'a>

pub fn new( text: &'a str, span: Range<usize>, char_span: Range<usize>, kind: TokenKind, ) -> Self

Construct a new Token.

Panics (debug only)

Panics in debug builds if span length does not match text.len(), or if char_span length does not match text.chars().count().

pub fn byte_len(&self) -> usize

Byte length of this token’s text.

pub fn char_len(&self) -> usize

Number of Unicode scalar values (chars) in this token’s text.

Trait Implementations

impl<'a> Clone for Token<'a>

fn clone(&self) -> Token<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Token<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> PartialEq for Token<'a>

fn eq(&self, other: &Token<'a>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a> Eq for Token<'a>

impl<'a> StructuralPartialEq for Token<'a>