Struct AssTokenizer

pub struct AssTokenizer<'a> { /* private fields */ }

Incremental tokenizer for ASS scripts with zero-copy design

Maintains lexical state for streaming tokenization. Uses &'a str spans to avoid allocations, with optional SIMD acceleration for hot paths.

Implementations

impl<'a> AssTokenizer<'a>

pub fn new(source: &'a str) -> Self

Create new tokenizer for source text

Handles BOM detection and UTF-8 validation upfront.

pub fn next_token(&mut self) -> Result<Option<Token<'a>>>

Get next token from input stream

Returns None when end of input reached. Maintains zero-copy semantics by returning spans into source text.

Errors

Returns an error if tokenization fails due to invalid input or scanner errors.

pub fn tokenize_all(&mut self) -> Result<Vec<Token<'a>>>

Get all tokens as vector for batch processing

Errors

Returns an error if tokenization fails for any token in the input.

pub fn issues(&self) -> &[TokenIssue<'a>]

Get accumulated tokenization issues

pub const fn position(&self) -> usize

Get current position in source

pub const fn line(&self) -> usize

Get current line number (1-based)

pub const fn column(&self) -> usize

Get current column number (1-based)

pub fn reset(&mut self)

Reset tokenizer to beginning of source

Trait Implementations

impl<'a> Clone for AssTokenizer<'a>

fn clone(&self) -> AssTokenizer<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for AssTokenizer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.