Enum TokenizeCommands 

pub enum TokenizeCommands {
    Plan {
        data: PathBuf,
        vocab_size: usize,
        algorithm: String,
        output: PathBuf,
        format: String,
    },
    Apply {
        data: PathBuf,
        vocab_size: usize,
        algorithm: String,
        output: PathBuf,
        max_lines: usize,
    },
    Train {
        corpus: PathBuf,
        vocab_size: usize,
        min_frequency: usize,
        output: PathBuf,
        normalization: String,
    },
    ImportHf {
        input: PathBuf,
        output: PathBuf,
        include_added_tokens: bool,
    },
    EncodeCorpus {
        corpus: PathBuf,
        tokenizer: PathBuf,
        output: PathBuf,
        shard_tokens: usize,
        content_field: String,
        normalization: String,
        eos_policy: String,
    },
}

Tokenizer training pipeline subcommands (forjar-style plan/apply).

Thin CLI wrappers around aprender’s BPE training infrastructure. Trains a BPE vocabulary from a text corpus for use in model training.

Variants

Plan

Validate inputs and estimate tokenizer training time/resources.

Checks that the input corpus exists, counts lines/bytes, estimates vocabulary coverage, and reports expected training time. Outputs a serializable plan manifest (text, JSON, or YAML).

Analogous to forjar plan — shows what will happen before committing.

Fields

data: PathBuf

Path to training corpus (text file, one document per line)

vocab_size: usize

Target vocabulary size

algorithm: String

Tokenizer algorithm: bpe, wordpiece, unigram

output: PathBuf

Output directory for trained tokenizer

format: String

Output format: text, json, yaml

Apply

Train a tokenizer on the corpus.

Reads the input corpus, trains a BPE/WordPiece/Unigram tokenizer, and writes vocab.json + merges.txt to the output directory.

Analogous to forjar apply — commits resources and executes the plan.

Fields

data: PathBuf

Path to training corpus (text file, one document per line)

vocab_size: usize

Target vocabulary size

algorithm: String

Tokenizer algorithm: bpe, wordpiece, unigram

output: PathBuf

Output directory for trained tokenizer

max_lines: usize

Maximum number of lines to read from corpus (0 = all)

Train

Train BPE on a JSONL corpus per contracts/tokenizer-bpe-v1.yaml (MODEL-2).

Walks --corpus (file or directory of .jsonl files), extracting the content field from each line, applies --normalization (NFC default), and trains a BPE tokenizer with the target vocab size. Writes vocab.json (token→id) and merges.txt (one a b pair per line, in merge order) to --output.

Fields

corpus: PathBuf

Path to corpus: a .jsonl file or a directory containing .jsonl files. Each line must be a JSON object with a content field.

vocab_size: usize

Target vocabulary size. Default 50_257 matches GPT-2 convention (50_000 BPE merges + 256 byte-level fallback tokens + 1 sentinel) and the MODEL-2 albor tokenizer contract (tokenizer-bpe-v1 v1.2.0).

min_frequency: usize

Minimum frequency a byte-pair must reach before BPE merges it into a new vocabulary token (honored by entrenar::tokenizer::BPETokenizer per task #103). Pairs below this threshold are left unmerged — contract INV-TOK-002 of contracts/tokenizer-bpe-v1.yaml.

output: PathBuf

Output directory; will contain vocab.json and merges.txt.

normalization: String

Unicode normalization form applied to each document before training.

ImportHf

Import a HuggingFace tokenizer.json into aprender’s two-file vocab.json + merges.txt layout per contracts/apr-cli-tokenize-import-hf-v1.yaml (§50.4 step 5g.0).

Reads <INPUT> (a HF tokenizer.json with model.type == "BPE"), extracts model.vocab → <OUTPUT>/vocab.json, model.merges → <OUTPUT>/merges.txt (one space-separated merge per line), and writes <OUTPUT>/manifest.json with extraction provenance (source path, sha256, vocab_size, merges_count, timestamp).

Non-BPE inputs (Unigram, WordPiece) are rejected fail-fast with a clear error citing the contract id.

Unblocks fine-tuning from public HF checkpoints (Qwen2.5/Llama2/ Mistral) which distribute as a single tokenizer.json. The output dir is consumable by apr tokenize encode-corpus --tokenizer <DIR> and apr pretrain --tokenizer <DIR> without modification.

Fields

input: PathBuf

Path to input HuggingFace tokenizer.json (BPE model required).

output: PathBuf

Output directory; will contain vocab.json + merges.txt + manifest.json.

include_added_tokens: bool

Include added_tokens in vocab.json (default: BPE state machine only). Use this when the downstream consumer needs special tokens (e.g., <|im_start|>, <|endoftext|>) materialized in vocab.json itself.

EncodeCorpus

Encode a JSONL corpus into .bin shards per contracts/pretokenize-bin-v1.yaml.

Loads a trained BPE tokenizer (vocab.json + merges.txt) from --tokenizer, reads --corpus (file or directory of .jsonl files), encodes the --content-field of each line to u32 tokens, and writes shard-NNNN.bin files (flat little-endian u32 streams) into --output. The output format is precisely what ShardBatchIter (aprender-train) expects at MODEL-2 pretrain read time.

Root-cause fix for the pretokenize-to-bin gap documented in memory/project_shard_reader_bin_format.md — replaces a Python shim that was flagged as MUDA on 2026-04-19.

Fields

corpus: PathBuf

Path to JSONL corpus file or directory of .jsonl files.

tokenizer: PathBuf

Directory containing vocab.json + merges.txt from apr tokenize train.

output: PathBuf

Output directory for shard-NNNN.bin + manifest.json.

shard_tokens: usize

Target tokens per shard (shard closes once this limit is reached).

content_field: String

JSONL field to encode (default: content).

normalization: String

Unicode normalization (must match tokenizer training).

eos_policy: String

EOS insertion policy: none|between|after.

Trait Implementations

impl Debug for TokenizeCommands

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromArgMatches for TokenizeCommands

fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn from_arg_matches_mut( __clap_arg_matches: &mut ArgMatches, ) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn update_from_arg_matches( &mut self, __clap_arg_matches: &ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

fn update_from_arg_matches_mut<'b>( &mut self, __clap_arg_matches: &mut ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

impl Subcommand for TokenizeCommands

fn augment_subcommands<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate Self via FromArgMatches::from_arg_matches_mut

fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate self via FromArgMatches::update_from_arg_matches_mut

fn has_subcommand(__clap_name: &str) -> bool

Test whether Self can parse a specific subcommand