Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Stateful, reusable line classifier. Buffer text with feed, flush the remainder with end, and reset with reset before reuse.

Implementations

impl Tokenizer

pub fn new() -> Self

Construct a fresh tokenizer at line 1.

pub fn feed(&mut self, chunk: &str) -> Vec<Token>

Feed a chunk and return all complete-line tokens. A partial trailing line (no trailing \n) is buffered until the next Self::feed or Self::end. No-op once Self::end has been called; call Self::reset to reuse.

pub fn end(&mut self) -> Vec<Token>

Flush any buffered partial line as a final token.

pub fn reset(&mut self)

Return to a fresh state for reuse.

pub fn tokenize_all(&mut self, text: &str) -> Vec<Token>

Tokenize an entire input in one shot.

pub fn tokenize(&self, line: &str, line_num: u32) -> Token

Classify a single line (no state).

pub fn is_op(&self, line: &str) -> bool

True when line parses as a hunk header verb.

pub fn is_header(&self, line: &str) -> bool

True when line parses as a [PATH(#HASH)?] header.

pub fn is_envelope_marker(&self, line: &str) -> bool

True when line is an envelope begin/end/abort marker.

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.