Struct spm_precompiled::Precompiled
source · [−]pub struct Precompiled { /* private fields */ }
Expand description
This struct is specifically done to be compatible with SentencePiece
SentencePiece models embed their Normalizer within a precompiled_charsmap
that both represents a Trie, and embedded rewrite rules.
In order to be 100% compliant we need to interpret that binary format too.
The format is [u32 (length of trie), trie: u32, normalized: String]
The trie has u8 as entries, and u32 as values, those u32 values
point to offsets withing the String that correspond to the real replace value
The normalized string contains ‘\0’ that should indicate the end of an entry.
Hence, normalized could be “abc\0”, some entry in the trie could be 0 meaning the value is “abc” and another one be 1 meaning the actual entry was “bc”.
Implementations
sourceimpl Precompiled
impl Precompiled
Trait Implementations
sourceimpl Clone for Precompiled
impl Clone for Precompiled
sourcefn clone(&self) -> Precompiled
fn clone(&self) -> Precompiled
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from source
. Read more
sourceimpl Debug for Precompiled
impl Debug for Precompiled
sourceimpl Default for Precompiled
impl Default for Precompiled
sourcefn default() -> Precompiled
fn default() -> Precompiled
Returns the “default value” for a type. Read more
sourceimpl<'de> Deserialize<'de> for Precompiled
impl<'de> Deserialize<'de> for Precompiled
sourcefn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
sourceimpl PartialEq<Precompiled> for Precompiled
impl PartialEq<Precompiled> for Precompiled
sourcefn eq(&self, other: &Precompiled) -> bool
fn eq(&self, other: &Precompiled) -> bool
This method tests for self
and other
values to be equal, and is used
by ==
. Read more
sourcefn ne(&self, other: &Precompiled) -> bool
fn ne(&self, other: &Precompiled) -> bool
This method tests for !=
.
sourceimpl Serialize for Precompiled
impl Serialize for Precompiled
impl StructuralPartialEq for Precompiled
Auto Trait Implementations
impl RefUnwindSafe for Precompiled
impl Send for Precompiled
impl Sync for Precompiled
impl Unpin for Precompiled
impl UnwindSafe for Precompiled
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
sourceimpl<T> ToOwned for T where
T: Clone,
impl<T> ToOwned for T where
T: Clone,
type Owned = T
type Owned = T
The resulting type after obtaining ownership.
sourcefn clone_into(&self, target: &mut T)
fn clone_into(&self, target: &mut T)
toowned_clone_into
)Uses borrowed data to replace owned data, usually by cloning. Read more