Struct spm_precompiled::Precompiled

pub struct Precompiled { /* private fields */ }

This struct is specifically done to be compatible with SentencePiece SentencePiece models embed their Normalizer within a precompiled_charsmap that both represents a Trie, and embedded rewrite rules. In order to be 100% compliant we need to interpret that binary format too. The format is [u32 (length of trie), trie: u32, normalized: String] The trie has u8 as entries, and u32 as values, those u32 values point to offsets withing the String that correspond to the real replace value The normalized string contains ‘\0’ that should indicate the end of an entry.

Hence, normalized could be “abc\0”, some entry in the trie could be 0 meaning the value is “abc” and another one be 1 meaning the actual entry was “bc”.

Implementations

impl Precompiled

pub fn from(
    precompiled_charsmap: &[u8]
) -> Result<Precompiled, PrecompiledError>

pub fn transform(&self, chunk: &str) -> Option<&str>

pub fn normalize_string(&self, original: &str) -> String

Trait Implementations

impl Clone for Precompiled

fn clone(&self) -> Precompiled

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Precompiled

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Precompiled

fn default() -> Precompiled

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Precompiled

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl PartialEq<Precompiled> for Precompiled

fn eq(&self, other: &Precompiled) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Precompiled) -> bool

This method tests for !=.

impl Serialize for Precompiled

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Precompiled