Struct tokenizers::pre_tokenizers::metaspace::Metaspace
source · pub struct Metaspace {
pub prepend_scheme: PrependScheme,
pub split: bool,
/* private fields */
}
Expand description
Replaces all the whitespaces by the provided meta character and then splits on this character
Fields§
§prepend_scheme: PrependScheme
§split: bool
Implementations§
source§impl Metaspace
impl Metaspace
pub fn new( replacement: char, prepend_scheme: PrependScheme, split: bool ) -> Self
pub fn get_replacement(&self) -> char
pub fn set_replacement(&mut self, replacement: char)
pub fn get_split(&self) -> bool
pub fn set_split(&mut self, split: bool)
pub fn get_prepend_scheme(&self) -> PrependScheme
pub fn set_prepend_scheme(&mut self, scheme: PrependScheme)
Trait Implementations§
source§impl<'de> Deserialize<'de> for Metaspace
impl<'de> Deserialize<'de> for Metaspace
source§fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>where
D: Deserializer<'de>,
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>where
D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
source§impl From<Metaspace> for DecoderWrapper
impl From<Metaspace> for DecoderWrapper
source§impl From<Metaspace> for PreTokenizerWrapper
impl From<Metaspace> for PreTokenizerWrapper
source§impl PartialEq for Metaspace
impl PartialEq for Metaspace
source§impl PreTokenizer for Metaspace
impl PreTokenizer for Metaspace
fn pre_tokenize(&self, pretokenized: &mut PreTokenizedString) -> Result<()>
impl Eq for Metaspace
impl StructuralPartialEq for Metaspace
Auto Trait Implementations§
impl Freeze for Metaspace
impl RefUnwindSafe for Metaspace
impl Send for Metaspace
impl Sync for Metaspace
impl Unpin for Metaspace
impl UnwindSafe for Metaspace
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read more