Enum TokenizeCommands 

pub enum TokenizeCommands {
    Plan {
        data: PathBuf,
        vocab_size: usize,
        algorithm: String,
        output: PathBuf,
        format: String,
    },
    Apply {
        data: PathBuf,
        vocab_size: usize,
        algorithm: String,
        output: PathBuf,
        max_lines: usize,
    },
}

Tokenizer training pipeline subcommands (forjar-style plan/apply).

Thin CLI wrappers around aprender’s BPE training infrastructure. Trains a BPE vocabulary from a text corpus for use in model training.

Variants

Plan

Validate inputs and estimate tokenizer training time/resources.

Checks that the input corpus exists, counts lines/bytes, estimates vocabulary coverage, and reports expected training time. Outputs a serializable plan manifest (text, JSON, or YAML).

Analogous to forjar plan — shows what will happen before committing.

Fields

data: PathBuf

Path to training corpus (text file, one document per line)

vocab_size: usize

Target vocabulary size

algorithm: String

Tokenizer algorithm: bpe, wordpiece, unigram

output: PathBuf

Output directory for trained tokenizer

format: String

Output format: text, json, yaml

Apply

Train a tokenizer on the corpus.

Reads the input corpus, trains a BPE/WordPiece/Unigram tokenizer, and writes vocab.json + merges.txt to the output directory.

Analogous to forjar apply — commits resources and executes the plan.

Fields

data: PathBuf

Path to training corpus (text file, one document per line)

vocab_size: usize

Target vocabulary size

algorithm: String

Tokenizer algorithm: bpe, wordpiece, unigram

output: PathBuf

Output directory for trained tokenizer

max_lines: usize

Maximum number of lines to read from corpus (0 = all)

Trait Implementations

impl Debug for TokenizeCommands

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromArgMatches for TokenizeCommands

fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn from_arg_matches_mut( __clap_arg_matches: &mut ArgMatches, ) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn update_from_arg_matches( &mut self, __clap_arg_matches: &ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

fn update_from_arg_matches_mut<'b>( &mut self, __clap_arg_matches: &mut ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

impl Subcommand for TokenizeCommands

fn augment_subcommands<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate Self via FromArgMatches::from_arg_matches_mut

fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate self via FromArgMatches::update_from_arg_matches_mut

fn has_subcommand(__clap_name: &str) -> bool

Test whether Self can parse a specific subcommand