pub struct ByteLevelTokenizer {
pub eos_token_id: u32,
pub bos_token_id: u32,
/* private fields */
}Expand description
A robust byte-level tokenizer with subword vocabulary lookup and lossless byte-fallback.
Fields§
§eos_token_id: u32§bos_token_id: u32Implementations§
Trait Implementations§
Source§impl Clone for ByteLevelTokenizer
impl Clone for ByteLevelTokenizer
Source§fn clone(&self) -> ByteLevelTokenizer
fn clone(&self) -> ByteLevelTokenizer
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read moreSource§impl Debug for ByteLevelTokenizer
impl Debug for ByteLevelTokenizer
Source§impl Default for ByteLevelTokenizer
impl Default for ByteLevelTokenizer
Source§fn default() -> ByteLevelTokenizer
fn default() -> ByteLevelTokenizer
Returns the “default value” for a type. Read more
Source§impl Tokenizer for ByteLevelTokenizer
impl Tokenizer for ByteLevelTokenizer
Source§fn encode(&self, text: &str) -> Vec<u32>
fn encode(&self, text: &str) -> Vec<u32>
Encodes a UTF-8 text string into a sequence of token IDs.
Source§fn decode(&self, tokens: &[u32]) -> String
fn decode(&self, tokens: &[u32]) -> String
Decodes a sequence of token IDs back into a UTF-8 string.
Source§fn vocab_size(&self) -> usize
fn vocab_size(&self) -> usize
Total vocabulary size.
Auto Trait Implementations§
impl Forget for ByteLevelTokenizer
impl Freeze for ByteLevelTokenizer
impl Move for ByteLevelTokenizer
impl RefUnwindSafe for ByteLevelTokenizer
impl Send for ByteLevelTokenizer
impl Sync for ByteLevelTokenizer
impl Unpin for ByteLevelTokenizer
impl UnsafeUnpin for ByteLevelTokenizer
impl UnwindSafe for ByteLevelTokenizer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more