Struct spm_precompiled::Precompiled[−][src]

pub struct Precompiled { /* fields omitted */ }

Expand description

This struct is specifically done to be compatible with SentencePiece SentencePiece models embed their Normalizer within a precompiled_charsmap that both represents a Trie, and embedded rewrite rules. In order to be 100% compliant we need to interpret that binary format too. The format is [u32 (length of trie), trie: u32, normalized: String] The trie has u8 as entries, and u32 as values, those u32 values point to offsets withing the String that correspond to the real replace value The normalized string contains ‘\0’ that should indicate the end of an entry.

Hence, normalized could be “abc\0”, some entry in the trie could be 0 meaning the value is “abc” and another one be 1 meaning the actual entry was “bc”.

Implementations

impl Precompiled

pub fn from(
    precompiled_charsmap: &[u8]
) -> Result<Precompiled, PrecompiledError>

pub fn transform(&self, chunk: &str) -> Option<&str>

pub fn normalize_string(&self, original: &str) -> String

Trait Implementations

impl Clone for Precompiled

fn clone(&self) -> Precompiled

Returns a copy of the value. Read more

1.0.0[src]

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

impl Debug for Precompiled

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

impl Default for Precompiled

fn default() -> Precompiled

Returns the “default value” for a type. Read more

impl<'de> Deserialize<'de> for Precompiled

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer. Read more

impl PartialEq<Precompiled> for Precompiled

fn eq(&self, other: &Precompiled) -> bool

This method tests for self and other values to be equal, and is used by ==. Read more

fn ne(&self, other: &Precompiled) -> bool

This method tests for !=.

impl Serialize for Precompiled

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer,

Serialize this value into the given Serde serializer. Read more

impl StructuralPartialEq for Precompiled

Auto Trait Implementations

impl RefUnwindSafe for Precompiled

impl Send for Precompiled

impl Sync for Precompiled

impl Unpin for Precompiled

impl UnwindSafe for Precompiled

Blanket Implementations

impl<T> Any for T where
    T: 'static + ?Sized,

pub fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more

impl<T> Borrow<T> for T where
    T: ?Sized,

pub fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more

impl<T> BorrowMut<T> for T where
    T: ?Sized,

pub fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more

impl<T> Conv for T

fn conv<T>(self) -> T where
    Self: Into<T>,

Converts self into T using Into<T>. Read more

impl<T> Conv for T

fn conv<T>(self) -> T where
    Self: Into<T>,

Converts self into a target type. Read more

impl<T> FmtForward for T

fn fmt_binary(self) -> FmtBinary<Self> where
    Self: Binary,

Causes self to use its Binary implementation when Debug-formatted.

fn fmt_display(self) -> FmtDisplay<Self> where
    Self: Display,

Causes self to use its Display implementation when Debug-formatted. Read more

fn fmt_lower_exp(self) -> FmtLowerExp<Self> where
    Self: LowerExp,

Causes self to use its LowerExp implementation when Debug-formatted. Read more

fn fmt_lower_hex(self) -> FmtLowerHex<Self> where
    Self: LowerHex,

Causes self to use its LowerHex implementation when Debug-formatted. Read more

fn fmt_octal(self) -> FmtOctal<Self> where
    Self: Octal,

Causes self to use its Octal implementation when Debug-formatted.

fn fmt_pointer(self) -> FmtPointer<Self> where
    Self: Pointer,

Causes self to use its Pointer implementation when Debug-formatted. Read more

fn fmt_upper_exp(self) -> FmtUpperExp<Self> where
    Self: UpperExp,

Causes self to use its UpperExp implementation when Debug-formatted. Read more

fn fmt_upper_hex(self) -> FmtUpperHex<Self> where
    Self: UpperHex,

Causes self to use its UpperHex implementation when Debug-formatted. Read more

impl<T> From<T> for T

pub fn from(t: T) -> T

Performs the conversion.

impl<T, U> Into<U> for T where
    U: From<T>,

pub fn into(self) -> U

Performs the conversion.

impl<T> Pipe for T where
    T: ?Sized,

fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> R

Pipes by value. This is generally the method you want to use. Read more

fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> R where
    R: 'a,

Borrows self and passes that borrow into the pipe function. Read more

fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> R where
    R: 'a,

Mutably borrows self and passes that borrow into the pipe function. Read more

fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R where
    Self: Borrow<B>,
    R: 'a,
    B: 'a + ?Sized,

Borrows self, then passes self.borrow() into the pipe function. Read more

fn pipe_borrow_mut<'a, B, R>(
    &'a mut self, 
    func: impl FnOnce(&'a mut B) -> R
) -> R where
    Self: BorrowMut<B>,
    R: 'a,
    B: 'a + ?Sized,

Mutably borrows self, then passes self.borrow_mut() into the pipe function. Read more

fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R where
    Self: AsRef<U>,
    U: 'a + ?Sized,
    R: 'a,

Borrows self, then passes self.as_ref() into the pipe function.

fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R where
    Self: AsMut<U>,
    U: 'a + ?Sized,
    R: 'a,

Mutably borrows self, then passes self.as_mut() into the pipe function. Read more

fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R where
    Self: Deref<Target = T>,
    T: 'a + ?Sized,
    R: 'a,

Borrows self, then passes self.deref() into the pipe function.

fn pipe_deref_mut<'a, T, R>(
    &'a mut self, 
    func: impl FnOnce(&'a mut T) -> R
) -> R where
    Self: DerefMut<Target = T> + Deref,
    T: 'a + ?Sized,
    R: 'a,

Mutably borrows self, then passes self.deref_mut() into the pipe function. Read more

impl<T> Pipe for T

fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> R

Pipes a value into a function that cannot ordinarily be called in suffix position. Read more

impl<T> PipeAsRef for T

fn pipe_as_ref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R where
    Self: AsRef<T>,
    T: 'a,
    R: 'a,

Pipes a trait borrow into a function that cannot normally be called in suffix position. Read more

fn pipe_as_mut<'a, T, R>(&'a mut self, func: impl FnOnce(&'a mut T) -> R) -> R where
    Self: AsMut<T>,
    T: 'a,
    R: 'a,

Pipes a trait mutable borrow into a function that cannot normally be called in suffix position. Read more

impl<T> PipeBorrow for T

fn pipe_borrow<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R where
    Self: Borrow<T>,
    T: 'a,
    R: 'a,

Pipes a trait borrow into a function that cannot normally be called in suffix position. Read more

fn pipe_borrow_mut<'a, T, R>(
    &'a mut self, 
    func: impl FnOnce(&'a mut T) -> R
) -> R where
    Self: BorrowMut<T>,
    T: 'a,
    R: 'a,

Pipes a trait mutable borrow into a function that cannot normally be called in suffix position. Read more

impl<T> PipeDeref for T

fn pipe_deref<'a, R>(&'a self, func: impl FnOnce(&'a Self::Target) -> R) -> R where
    Self: Deref,
    R: 'a,

Pipes a dereference into a function that cannot normally be called in suffix position. Read more

fn pipe_deref_mut<'a, R>(
    &'a mut self, 
    func: impl FnOnce(&'a mut Self::Target) -> R
) -> R where
    Self: DerefMut,
    R: 'a,

Pipes a mutable dereference into a function that cannot normally be called in suffix position. Read more

impl<T> PipeRef for T

fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> R where
    R: 'a,

Pipes a reference into a function that cannot ordinarily be called in suffix position. Read more

fn pipe_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> R where
    R: 'a,

Pipes a mutable reference into a function that cannot ordinarily be called in suffix position. Read more

impl<T> Tap for T

fn tap(self, func: impl FnOnce(&Self)) -> Self

Immutable access to a value. Read more

fn tap_mut(self, func: impl FnOnce(&mut Self)) -> Self

Mutable access to a value. Read more

fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self where
    Self: Borrow<B>,
    B: ?Sized,

Immutable access to the Borrow<B> of a value. Read more

fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self where
    Self: BorrowMut<B>,
    B: ?Sized,

Mutable access to the BorrowMut<B> of a value. Read more

fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self where
    Self: AsRef<R>,
    R: ?Sized,

Immutable access to the AsRef<R> view of a value. Read more

fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self where
    Self: AsMut<R>,
    R: ?Sized,

Mutable access to the AsMut<R> view of a value. Read more

fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self where
    Self: Deref<Target = T>,
    T: ?Sized,

Immutable access to the Deref::Target of a value. Read more

fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self where
    Self: DerefMut<Target = T> + Deref,
    T: ?Sized,

Mutable access to the Deref::Target of a value. Read more

fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self

Calls .tap() only in debug builds, and is erased in release builds.

fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self

Calls .tap_mut() only in debug builds, and is erased in release builds. Read more

fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self where
    Self: Borrow<B>,
    B: ?Sized,

Calls .tap_borrow() only in debug builds, and is erased in release builds. Read more

fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self where
    Self: BorrowMut<B>,
    B: ?Sized,

Calls .tap_borrow_mut() only in debug builds, and is erased in release builds. Read more

fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self where
    Self: AsRef<R>,
    R: ?Sized,

Calls .tap_ref() only in debug builds, and is erased in release builds. Read more

fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self where
    Self: AsMut<R>,
    R: ?Sized,

Calls .tap_ref_mut() only in debug builds, and is erased in release builds. Read more

fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self where
    Self: Deref<Target = T>,
    T: ?Sized,

Calls .tap_deref() only in debug builds, and is erased in release builds. Read more

fn tap_deref_mut_dbg<T>(self, func: impl FnOnce(&mut T)) -> Self where
    Self: DerefMut<Target = T> + Deref,
    T: ?Sized,

Calls .tap_deref_mut() only in debug builds, and is erased in release builds. Read more

impl<T> Tap for T

fn tap<F, R>(self, func: F) -> Self where
    F: FnOnce(&Self) -> R,

Provides immutable access for inspection. Read more

fn tap_dbg<F, R>(self, func: F) -> Self where
    F: FnOnce(&Self) -> R,

Calls tap in debug builds, and does nothing in release builds.

fn tap_mut<F, R>(self, func: F) -> Self where
    F: FnOnce(&mut Self) -> R,

Provides mutable access for modification. Read more

fn tap_mut_dbg<F, R>(self, func: F) -> Self where
    F: FnOnce(&mut Self) -> R,

Calls tap_mut in debug builds, and does nothing in release builds.

impl<T, U> TapAsRef<U> for T where
    U: ?Sized,

fn tap_ref<F, R>(self, func: F) -> Self where
    Self: AsRef<T>,
    F: FnOnce(&T) -> R,

Provides immutable access to the reference for inspection.

fn tap_ref_dbg<F, R>(self, func: F) -> Self where
    Self: AsRef<T>,
    F: FnOnce(&T) -> R,

Calls tap_ref in debug builds, and does nothing in release builds.

fn tap_ref_mut<F, R>(self, func: F) -> Self where
    Self: AsMut<T>,
    F: FnOnce(&mut T) -> R,

Provides mutable access to the reference for modification.

fn tap_ref_mut_dbg<F, R>(self, func: F) -> Self where
    Self: AsMut<T>,
    F: FnOnce(&mut T) -> R,

Calls tap_ref_mut in debug builds, and does nothing in release builds.

impl<T, U> TapBorrow<U> for T where
    U: ?Sized,

fn tap_borrow<F, R>(self, func: F) -> Self where
    Self: Borrow<T>,
    F: FnOnce(&T) -> R,

Provides immutable access to the borrow for inspection. Read more

fn tap_borrow_dbg<F, R>(self, func: F) -> Self where
    Self: Borrow<T>,
    F: FnOnce(&T) -> R,

Calls tap_borrow in debug builds, and does nothing in release builds.

fn tap_borrow_mut<F, R>(self, func: F) -> Self where
    Self: BorrowMut<T>,
    F: FnOnce(&mut T) -> R,

Provides mutable access to the borrow for modification.

fn tap_borrow_mut_dbg<F, R>(self, func: F) -> Self where
    Self: BorrowMut<T>,
    F: FnOnce(&mut T) -> R,

Calls tap_borrow_mut in debug builds, and does nothing in release builds. Read more

impl<T> TapDeref for T

fn tap_deref<F, R>(self, func: F) -> Self where
    Self: Deref,
    F: FnOnce(&Self::Target) -> R,

Immutably dereferences self for inspection.

fn tap_deref_dbg<F, R>(self, func: F) -> Self where
    Self: Deref,
    F: FnOnce(&Self::Target) -> R,

Calls tap_deref in debug builds, and does nothing in release builds.

fn tap_deref_mut<F, R>(self, func: F) -> Self where
    Self: DerefMut,
    F: FnOnce(&mut Self::Target) -> R,

Mutably dereferences self for modification.

fn tap_deref_mut_dbg<F, R>(self, func: F) -> Self where
    Self: DerefMut,
    F: FnOnce(&mut Self::Target) -> R,

Calls tap_deref_mut in debug builds, and does nothing in release builds. Read more

impl<T> ToOwned for T where
    T: Clone,

type Owned = T

The resulting type after obtaining ownership.

pub fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more

pub fn clone_into(&self, target: &mut T)

🔬 This is a nightly-only experimental API. (toowned_clone_into)

recently added

Uses borrowed data to replace owned data, usually by cloning. Read more

impl<T> TryConv for T

fn try_conv<T>(self) -> Result<T, Self::Error> where
    Self: TryInto<T>,

Attempts to convert self into T using TryInto<T>. Read more

impl<T> TryConv for T

fn try_conv<T>(self) -> Result<T, Self::Error> where
    Self: TryInto<T>,

Attempts to convert self into a target type. Read more

impl<T, U> TryFrom<U> for T where
    U: Into<T>,

type Error = Infallible

The type returned in the event of a conversion error.

pub fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.

impl<T, U> TryInto<U> for T where
    U: TryFrom<T>,

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.

pub fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.

impl<T> DeserializeOwned for T where
    T: for<'de> Deserialize<'de>,