Struct finalfrontier::SubwordVocabConfig

pub struct SubwordVocabConfig {
    pub min_n: u32,
    pub max_n: u32,
    pub buckets_exp: u32,
    pub min_count: u32,
    pub discard_threshold: f32,
}

Hyperparameters for subword-vocabs.

Fields

min_n: u32

Minimum n-gram length for subword units (inclusive).

max_n: u32

Maximum n-gram length for subword units (inclusive).

buckets_exp: u32

Bucket exponent. The model will use 2^bucket_exp buckets.

A typical value for this parameter is 21, which gives roughly 2M buckets.

min_count: u32

Minimum token count.

No word-specific embeddings will be trained for tokens occurring less than this count.

discard_threshold: f32

Discard threshold.

The discard threshold is used to compute the discard probability of a token. E.g. with a threshold of 0.00001 tokens with approximately that probability will never be discarded.

Trait Implementations

impl Copy for SubwordVocabConfig

impl Clone for SubwordVocabConfig

fn clone(&self) -> SubwordVocabConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SubwordVocabConfig

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Serialize for SubwordVocabConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.