pub trait Tokenize {
// Required method
fn tokenize(&self, output: &mut CellStream) -> Result<(), CellStreamError>;
// Provided method
fn to_cell_stream(&self) -> Result<CellStream, CellStreamError> { ... }
}Expand description
Converts syntax or raw tokens into structural cells.
This boundary is intentionally distinct from Lex: most macros should
preserve cell partitioning, while function-like expansion may explicitly
choose a raw TokenStream when syntax is not valid before expansion.
Required Methods§
Sourcefn tokenize(&self, output: &mut CellStream) -> Result<(), CellStreamError>
fn tokenize(&self, output: &mut CellStream) -> Result<(), CellStreamError>
Appends this value’s independently parseable cells to output.
§Errors
Returns CellStreamError when a raw token prefix does not end at a
complete cell boundary.
Provided Methods§
Sourcefn to_cell_stream(&self) -> Result<CellStream, CellStreamError>
fn to_cell_stream(&self) -> Result<CellStream, CellStreamError>
Tokenizes this value into a fresh cell stream.
§Errors
Returns CellStreamError when a raw token prefix does not end at a
complete cell boundary.
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".