pub enum TokenType {
Normal,
Unknown,
Control,
UserDefined,
Unused,
Byte,
Other(i32),
}Expand description
A token classification from tokenizer.ggml.token_type metadata.
Variants§
Normal
A normal vocabulary token.
Unknown
The tokenizer’s unknown-token marker.
Control
A control token.
UserDefined
A user-defined token.
Unused
A vocabulary entry reserved as unused.
Byte
A token representing a raw byte.
Other(i32)
An unrecognized GGUF token-type code.
Trait Implementations§
impl Copy for TokenType
impl Eq for TokenType
impl StructuralPartialEq for TokenType
Auto Trait Implementations§
impl Forget for TokenType
impl Freeze for TokenType
impl Move for TokenType
impl RefUnwindSafe for TokenType
impl Send for TokenType
impl Sync for TokenType
impl Unpin for TokenType
impl UnsafeUnpin for TokenType
impl UnwindSafe for TokenType
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more