pub struct ByteTokenizer;Implementations§
Source§impl ByteTokenizer
impl ByteTokenizer
pub const VOCAB_SIZE: usize = 256
pub fn encode(text: &str) -> Vec<u32>
Sourcepub fn decode(ids: &[u32]) -> String
pub fn decode(ids: &[u32]) -> String
Decodes token ids back to a string. Ids outside 0..256 are dropped rather than silently corrupting output; invalid UTF-8 byte sequences are replaced per Rust’s standard lossy conversion.
Sourcepub fn decode_bytes(ids: &[u32]) -> Vec<u8> ⓘ
pub fn decode_bytes(ids: &[u32]) -> Vec<u8> ⓘ
The raw bytes, before any UTF-8 decision is made about them.
A caller decoding ONE token at a time must have these: a
multi-byte character split across two tokens is two invalid
fragments, and decode would turn each into U+FFFD and lose the
bytes for good. See ferrox_server::utf8_stream.
Auto Trait Implementations§
impl Freeze for ByteTokenizer
impl RefUnwindSafe for ByteTokenizer
impl Send for ByteTokenizer
impl Sync for ByteTokenizer
impl Unpin for ByteTokenizer
impl UnsafeUnpin for ByteTokenizer
impl UnwindSafe for ByteTokenizer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more