pub struct EstimateTokenCounter;Expand description
Default token counter: estimates tokens as len / 4.
Acceptable for English prose (~4 chars per token on average). Inaccurate for CJK text (~1 token per char), code, or structured data. Replace with a real tokenizer for accurate budget management.
Trait Implementations§
Source§impl TokenCounter for EstimateTokenCounter
impl TokenCounter for EstimateTokenCounter
Source§fn count_tokens(&self, text: &str) -> usize
fn count_tokens(&self, text: &str) -> usize
Count the number of tokens in the given text.
Auto Trait Implementations§
impl Freeze for EstimateTokenCounter
impl RefUnwindSafe for EstimateTokenCounter
impl Send for EstimateTokenCounter
impl Sync for EstimateTokenCounter
impl Unpin for EstimateTokenCounter
impl UnsafeUnpin for EstimateTokenCounter
impl UnwindSafe for EstimateTokenCounter
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more