pub struct FasterHuggingFaceTokenizer { /* private fields */ }
Expand description
A tokenizer that uses the HuggingFace tokenizer with a cache for single tokens.
Implementations§
source§impl FasterHuggingFaceTokenizer
impl FasterHuggingFaceTokenizer
sourcepub fn new(tokenizer: Tokenizer) -> Self
pub fn new(tokenizer: Tokenizer) -> Self
Create a new FasterHuggingFaceTokenizer
from a tokenizers::Tokenizer
.
sourcepub fn tokenizer_mut(&mut self) -> &mut Tokenizer
pub fn tokenizer_mut(&mut self) -> &mut Tokenizer
Get the inner tokenizer mutably.
sourcepub fn into_tokenizer(self) -> Tokenizer
pub fn into_tokenizer(self) -> Tokenizer
Consume the FasterHuggingFaceTokenizer
and return the inner tokenizer.
Trait Implementations§
source§impl Tokenizer for FasterHuggingFaceTokenizer
impl Tokenizer for FasterHuggingFaceTokenizer
source§fn encode(&self, text: &str, special_tokens: bool) -> Result<Vec<u32>>
fn encode(&self, text: &str, special_tokens: bool) -> Result<Vec<u32>>
Encode a string into a list of token ids.
source§fn decode(&self, ids: &[u32]) -> Result<Cow<'_, str>>
fn decode(&self, ids: &[u32]) -> Result<Cow<'_, str>>
Decode a list of token ids into a string.
Auto Trait Implementations§
impl RefUnwindSafe for FasterHuggingFaceTokenizer
impl Send for FasterHuggingFaceTokenizer
impl Sync for FasterHuggingFaceTokenizer
impl Unpin for FasterHuggingFaceTokenizer
impl UnwindSafe for FasterHuggingFaceTokenizer
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more