Enum TokenizeCommands 

pub enum TokenizeCommands {
    Plan {
        data: PathBuf,
        vocab_size: usize,
        algorithm: String,
        output: PathBuf,
        format: String,
    },
    Apply {
        data: PathBuf,
        vocab_size: usize,
        algorithm: String,
        output: PathBuf,
        max_lines: usize,
    },
    Train {
        corpus: PathBuf,
        vocab_size: usize,
        min_frequency: usize,
        output: PathBuf,
        normalization: String,
    },
    ImportHf {
        input: PathBuf,
        output: PathBuf,
        include_added_tokens: bool,
    },
    EncodeCorpus {

        corpus: Vec<PathBuf>,
        tokenizer: PathBuf,
        output: PathBuf,
        shard_tokens: usize,
        content_field: String,
        normalization: String,
        eos_policy: String,
        num_workers: Option<usize>,
        quiet: bool,
        progress_interval_docs: u64,
        progress_interval_seconds: u64,
        estimate_only: bool,
        estimate_sample_docs: u64,
    
},
    RepairManifest {
        output: PathBuf,
        tokenizer: Option<PathBuf>,
        json: bool,
    },
}

Tokenizer training pipeline subcommands (forjar-style plan/apply).

Thin CLI wrappers around aprender’s BPE training infrastructure. Trains a BPE vocabulary from a text corpus for use in model training.

Variants

Plan

Validate inputs and estimate tokenizer training time/resources.

Checks that the input corpus exists, counts lines/bytes, estimates vocabulary coverage, and reports expected training time. Outputs a serializable plan manifest (text, JSON, or YAML).

Analogous to forjar plan — shows what will happen before committing.

Fields

data: PathBuf

Path to training corpus (text file, one document per line)

vocab_size: usize

Target vocabulary size

algorithm: String

Tokenizer algorithm: bpe, wordpiece, unigram

output: PathBuf

Output directory for trained tokenizer

format: String

Output format: text, json, yaml

Apply

Train a tokenizer on the corpus.

Reads the input corpus, trains a BPE/WordPiece/Unigram tokenizer, and writes vocab.json + merges.txt to the output directory.

Analogous to forjar apply — commits resources and executes the plan.

Fields

data: PathBuf

Path to training corpus (text file, one document per line)

vocab_size: usize

Target vocabulary size

algorithm: String

Tokenizer algorithm: bpe, wordpiece, unigram

output: PathBuf

Output directory for trained tokenizer

max_lines: usize

Maximum number of lines to read from corpus (0 = all)

Train

Train BPE on a JSONL corpus per contracts/tokenizer-bpe-v1.yaml (MODEL-2).

Walks --corpus (file or directory of .jsonl files), extracting the content field from each line, applies --normalization (NFC default), and trains a BPE tokenizer with the target vocab size. Writes vocab.json (token→id) and merges.txt (one a b pair per line, in merge order) to --output.

Fields

corpus: PathBuf

Path to corpus: a .jsonl file or a directory containing .jsonl files. Each line must be a JSON object with a content field.

vocab_size: usize

Target vocabulary size. Default 50_257 matches GPT-2 convention (50_000 BPE merges + 256 byte-level fallback tokens + 1 sentinel) and the MODEL-2 albor tokenizer contract (tokenizer-bpe-v1 v1.2.0).

min_frequency: usize

Minimum frequency a byte-pair must reach before BPE merges it into a new vocabulary token (honored by entrenar::tokenizer::BPETokenizer per task #103). Pairs below this threshold are left unmerged — contract INV-TOK-002 of contracts/tokenizer-bpe-v1.yaml.

output: PathBuf

Output directory; will contain vocab.json and merges.txt.

normalization: String

Unicode normalization form applied to each document before training.

ImportHf

Import a HuggingFace tokenizer.json into aprender’s two-file vocab.json + merges.txt layout per contracts/apr-cli-tokenize-import-hf-v1.yaml (§50.4 step 5g.0).

Reads <INPUT> (a HF tokenizer.json with model.type == "BPE"), extracts model.vocab → <OUTPUT>/vocab.json, model.merges → <OUTPUT>/merges.txt (one space-separated merge per line), and writes <OUTPUT>/manifest.json with extraction provenance (source path, sha256, vocab_size, merges_count, timestamp).

Non-BPE inputs (Unigram, WordPiece) are rejected fail-fast with a clear error citing the contract id.

Unblocks fine-tuning from public HF checkpoints (Qwen2.5/Llama2/ Mistral) which distribute as a single tokenizer.json. The output dir is consumable by apr tokenize encode-corpus --tokenizer <DIR> and apr pretrain --tokenizer <DIR> without modification.

Fields

input: PathBuf

Path to input HuggingFace tokenizer.json (BPE model required).

output: PathBuf

Output directory; will contain vocab.json + merges.txt + manifest.json.

include_added_tokens: bool

Include added_tokens in vocab.json (default: BPE state machine only). Use this when the downstream consumer needs special tokens (e.g., <|im_start|>, <|endoftext|>) materialized in vocab.json itself.

EncodeCorpus

Encode a JSONL corpus into .bin shards per contracts/pretokenize-bin-v1.yaml.

Loads a trained BPE tokenizer (vocab.json + merges.txt) from --tokenizer, reads --corpus (file or directory of .jsonl files), encodes the --content-field of each line to u32 tokens, and writes shard-NNNN.bin files (flat little-endian u32 streams) into --output. The output format is precisely what ShardBatchIter (aprender-train) expects at MODEL-2 pretrain read time.

Root-cause fix for the pretokenize-to-bin gap documented in memory/project_shard_reader_bin_format.md — replaces a Python shim that was flagged as MUDA on 2026-04-19.

Fields

corpus: Vec<PathBuf>

Path to JSONL corpus file, parquet shard, or directory of .jsonl or .parquet files. Pass --corpus multiple times to merge multiple sources into a single output corpus (SPEC §83 P2-C — see contracts/corpus-merge-v3-v1.yaml). When repeated, sources are encoded in command-line order and shard numbering is continuous across sources.

tokenizer: PathBuf

Directory containing vocab.json + merges.txt from apr tokenize train.

output: PathBuf

Output directory for shard-NNNN.bin + manifest.json.

shard_tokens: usize

Target tokens per shard (shard closes once this limit is reached).

content_field: String

JSONL field to encode (default: content).

normalization: String

Unicode normalization (must match tokenizer training).

eos_policy: String

EOS insertion policy: none|between|after.

num_workers: Option<usize>

Number of rayon workers for per-document BPE encoding.

Defaults to std::thread::available_parallelism() (logical CPU count). Set to 1 to force the single-threaded byte-identical legacy path. Set to a fixed N to bound memory or share the host with other jobs.

Output shard order is preserved: chunked encoding keeps original document order regardless of worker count (issue #1547, contracts/apr-tokenize-parallel-bpe-v1.yaml parallel_correctness).

quiet: bool

Suppress per-document progress emission to stderr (issue #1547, contract v1.2.0). Default: emit a [progress] doc=N/T tokens=K rate=X.X docs/s eta=... line every --progress-interval-docs docs OR --progress-interval-seconds seconds (whichever fires first). Useful for CI / log-scraping callers that prefer silence.

progress_interval_docs: u64

Emit a progress line at most every N docs (default 1000). Pair with --progress-interval-seconds — whichever bound is reached first triggers emission. Issue #1547 contract v1.2.0.

progress_interval_seconds: u64

Emit a progress line at most every S seconds (default 60). Pair with --progress-interval-docs — whichever bound is reached first triggers emission. Issue #1547 contract v1.2.0.

estimate_only: bool

Pre-flight only: estimate total tokens / shards / wall time without writing any output. Reads --estimate-sample-docs (default 1000), encodes them, observes (tokens, wall-time- per-doc), and extrapolates against the total document count. Emits [estimate] lines on stderr; no shards or manifest are written. Operator pre-flight gate before multi-day encode runs (issue #1547 contract v1.3.0).

estimate_sample_docs: u64

Number of documents to sample for --estimate-only extrapolation (default: 1000). Larger samples → tighter per-doc rate estimate but longer pre-flight wall.

RepairManifest

Reconstruct manifest.json from existing shard-NNNN.bin files.

apr tokenize encode-corpus writes manifest.json only on clean process exit. If the encoder is killed (operator SIGINT, OOM, crash, power loss) AFTER all shards flush but BEFORE manifest write, the corpus on disk is consumable by ShardBatchIter but has no provenance file for ship audit / dashboards.

repair-manifest is the cheap recovery path: it scans <OUTPUT>/shard-*.bin, computes shard_count + total_tokens from file sizes (each shard is a flat little-endian u32 stream; tokens = file_size / 4), and writes a schema-conforming manifest.json. Idempotent: runs twice are byte-identical modulo repaired_at timestamp.

Contract: contracts/apr-tokenize-repair-manifest-v1.yaml. Motivating instance: SHIP-TWO §56 5g.1 corpus (228 shards flushed, manifest missing).

Fields

output: PathBuf

Output directory containing shard-NNNN.bin files. manifest.json will be written into this directory.

tokenizer: Option<PathBuf>

Optional tokenizer directory; when provided, vocab.json is read for the manifest’s vocab_size field. Without it, vocab_size is recorded as null (provenance-incomplete but otherwise valid).

json: bool

Emit the manifest body as JSON to stdout (in addition to writing to disk).

Trait Implementations

impl Debug for TokenizeCommands

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromArgMatches for TokenizeCommands

fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn from_arg_matches_mut( __clap_arg_matches: &mut ArgMatches, ) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn update_from_arg_matches( &mut self, __clap_arg_matches: &ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

fn update_from_arg_matches_mut<'b>( &mut self, __clap_arg_matches: &mut ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

impl Subcommand for TokenizeCommands

fn augment_subcommands<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate Self via FromArgMatches::from_arg_matches_mut

fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate self via FromArgMatches::update_from_arg_matches_mut

fn has_subcommand(__clap_name: &str) -> bool

Test whether Self can parse a specific subcommand