Trait Tokenize

Source

pub trait Tokenize {
    // Required method
    fn tokenize(&self, output: &mut CellStream) -> Result<(), CellStreamError>;

    // Provided method
    fn to_cell_stream(&self) -> Result<CellStream, CellStreamError> { ... }
}

Expand description

Converts syntax or raw tokens into structural cells.

This boundary is intentionally distinct from Lex: most macros should preserve cell partitioning, while function-like expansion may explicitly choose a raw TokenStream when syntax is not valid before expansion. Implementations append to the supplied output rather than replacing its existing cells.

§Examples

use m2_syn::{CellStream, Tokenize, lex_str};

let first = lex_str("x = 1;")?;
let second = lex_str("y = 2")?;
let mut cells = CellStream::default();

first.tokenize(&mut cells)?;
second.tokenize(&mut cells)?;

assert_eq!(cells.len(), 2);