pub struct ByteTokenizer { /* private fields */ }Expand description
Byte-level tokenizer: token id = byte value; specials live above 255.
Implementations§
Source§impl ByteTokenizer
impl ByteTokenizer
Sourcepub fn new(vocab_size: usize) -> Result<Self>
pub fn new(vocab_size: usize) -> Result<Self>
Create a tokenizer; vocab_size must be >= MIN_VOCAB.
The byte-level scheme is fixed at MIN_VOCAB ids: with a larger
vocab_size, ids 258..vocab_size are never produced by
ByteTokenizer::encode (dead ids reserved for a future text-level
vocabulary extension).
Sourcepub fn encode(&self, text: &str, with_bos: bool, with_eos: bool) -> Vec<u32>
pub fn encode(&self, text: &str, with_bos: bool, with_eos: bool) -> Vec<u32>
Encode text to token ids, optionally adding BOS/EOS.
Sourcepub fn decode(&self, tokens: &[u32]) -> String
pub fn decode(&self, tokens: &[u32]) -> String
Decode token ids back to text. Special tokens (ids >= 256) are skipped. Bytes that do not form valid UTF-8 decode as the replacement character (lossy decoding).
Sourcepub fn vocab_size(&self) -> usize
pub fn vocab_size(&self) -> usize
Vocabulary size this tokenizer was built for.
Trait Implementations§
Source§impl Clone for ByteTokenizer
impl Clone for ByteTokenizer
Auto Trait Implementations§
impl Forget for ByteTokenizer
impl Freeze for ByteTokenizer
impl RefUnwindSafe for ByteTokenizer
impl Send for ByteTokenizer
impl Sync for ByteTokenizer
impl Unpin for ByteTokenizer
impl UnsafeUnpin for ByteTokenizer
impl UnwindSafe for ByteTokenizer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> ErasedDestructor for Twhere
T: 'static,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more