pub enum Truncation {
Reject(usize),
Right(usize),
}Expand description
Explicit right-truncation policy. Truncation occurs after special-token insertion and can remove trailing special tokens; use tokenizer-native postprocessing when a model requires preserving a closing separator.
Variants§
Reject(usize)
Reject sequences longer than this limit.
Right(usize)
Keep at most the first specified number of tokens.
Trait Implementations§
Source§impl Clone for Truncation
impl Clone for Truncation
impl Copy for Truncation
Auto Trait Implementations§
impl Forget for Truncation
impl Freeze for Truncation
impl RefUnwindSafe for Truncation
impl Send for Truncation
impl Sync for Truncation
impl Unpin for Truncation
impl UnsafeUnpin for Truncation
impl UnwindSafe for Truncation
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more