pub enum TokenEstimator {
Simd(SimdWhitespaceTokenizer),
}Expand description
Enum that dispatches to the best available token counter.
With the tiktoken feature enabled (on non-WASM targets) this wraps a
[TiktokenCounter]; otherwise it uses SimdWhitespaceTokenizer.
Variants§
Simd(SimdWhitespaceTokenizer)
Fast SWAR-accelerated approximation.
Implementations§
Trait Implementations§
Source§impl Debug for TokenEstimator
impl Debug for TokenEstimator
Source§impl Default for TokenEstimator
impl Default for TokenEstimator
Source§impl TokenCounter for TokenEstimator
impl TokenCounter for TokenEstimator
Auto Trait Implementations§
impl Freeze for TokenEstimator
impl RefUnwindSafe for TokenEstimator
impl Send for TokenEstimator
impl Sync for TokenEstimator
impl Unpin for TokenEstimator
impl UnsafeUnpin for TokenEstimator
impl UnwindSafe for TokenEstimator
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more