Struct finalfrontier::SubwordVocabConfig

pub struct SubwordVocabConfig<V> {
    pub min_count: u32,
    pub discard_threshold: f32,
    pub min_n: u32,
    pub max_n: u32,
    pub indexer: V,
}

Hyperparameters for Subword vocabs.

Fields

min_count: u32

Minimum token count.

No word-specific embeddings will be trained for tokens occurring less than this count.

discard_threshold: f32

Discard threshold.

The discard threshold is used to compute the discard probability of a token. E.g. with a threshold of 0.00001 tokens with approximately that probability will never be discarded.

min_n: u32

Minimum n-gram length for subword units (inclusive).

max_n: u32

Maximum n-gram length for subword units (inclusive).

indexer: V

Indexer specific parameters.

Trait Implementations

impl<V: Clone> Clone for SubwordVocabConfig<V>

fn clone(&self) -> SubwordVocabConfig<V>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<V: Copy> Copy for SubwordVocabConfig<V>

impl<V: Debug> Debug for SubwordVocabConfig<V>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<V> Serialize for SubwordVocabConfig<V> where
    V: Serialize, 

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.