Struct TokenStream 

pub struct TokenStream { /* private fields */ }

Streaming token counter.

Implementations

impl TokenStream

pub fn new() -> Self

Construct a stream using the default 4-chars-per-token estimator.

pub fn with_estimator<F>(est: F) -> Self

where F: FnMut(&str) -> u64 + Send + 'static,

Construct a stream that runs est on each pushed chunk.

pub fn push(&mut self, chunk: &str) -> u64

Push the next chunk. Returns the new running count.

pub fn count(&self) -> u64

Current running token count.

pub fn chars(&self) -> u64

Total characters pushed so far.

pub fn reset(&mut self)

Reset the counter (estimator preserved).

Trait Implementations

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenStream

fn default() -> Self

Returns the “default value” for a type.