Trait Trainer 

pub trait Trainer {
    type Model: Model + Sized;

    // Required methods
    fn should_show_progress(&self) -> bool;
    fn train(&self, model: &mut Self::Model) -> Result<Vec<AddedToken>>;
    fn feed<I, S, F>(&mut self, iterator: I, process: F) -> Result<()>
       where I: Iterator<Item = S> + Send,
             S: AsRef<str> + Send,
             F: Fn(&str) -> Result<Vec<String>> + Sync;
}

A Trainer has the responsibility to train a model. We feed it with lines/sentences and then it can train the given Model.

Required Associated Types

type Model: Model + Sized

Required Methods

fn should_show_progress(&self) -> bool

Whether we should show progress during the training.

fn train(&self, model: &mut Self::Model) -> Result<Vec<AddedToken>>

The actual training method. This will return a new trained Model as well as a list of special_tokens to be added directly to the tokenizer along with the model.

fn feed<I, S, F>(&mut self, iterator: I, process: F) -> Result<()>

where I: Iterator<Item = S> + Send, S: AsRef<str> + Send, F: Fn(&str) -> Result<Vec<String>> + Sync,

Process an iterator of sequences, calling process for each of them in order to pre-process the said sequence as relevant.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl Trainer for TrainerWrapper

type Model = ModelWrapper

impl Trainer for BpeTrainer

type Model = BPE

impl Trainer for UnigramTrainer

type Model = Unigram

impl Trainer for WordLevelTrainer

type Model = WordLevel

impl Trainer for WordPieceTrainer

type Model = WordPiece