pub enum PostProcessorConfig {
Sequence {
processors: Vec<PostProcessorConfig>,
},
ByteLevel {
add_prefix_space: bool,
trim_offsets: bool,
use_regex: bool,
},
TemplateProcessing {
single: Value,
pair: Value,
special_tokens: Value,
},
}Expand description
Post-processor pipeline step from tokenizer.json.
Variants§
Trait Implementations§
Source§impl Clone for PostProcessorConfig
impl Clone for PostProcessorConfig
Source§impl Debug for PostProcessorConfig
impl Debug for PostProcessorConfig
Source§impl<'de> Deserialize<'de> for PostProcessorConfig
impl<'de> Deserialize<'de> for PostProcessorConfig
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
Auto Trait Implementations§
impl Freeze for PostProcessorConfig
impl RefUnwindSafe for PostProcessorConfig
impl Send for PostProcessorConfig
impl Sync for PostProcessorConfig
impl Unpin for PostProcessorConfig
impl UnsafeUnpin for PostProcessorConfig
impl UnwindSafe for PostProcessorConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> DeserializeOwned for Twhere
T: for<'de> Deserialize<'de>,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more