Enum Model 

pub enum Model {
    OpenAiGpt4,
    OpenAiGpt4o,
    AnthropicClaude,
    Heuristic,
}

A popular agentic AI system, identified by its tokenizer family.

Variants

OpenAiGpt4

OpenAI GPT-4 / GPT-3.5-turbo family — cl100k_base BPE.

OpenAiGpt4o

OpenAI GPT-4o / o-series family — o200k_base BPE.

AnthropicClaude

Anthropic Claude. Approximation: Anthropic publishes no offline tokenizer crate, so this uses the shared heuristic_tokens estimate (the same as Model::Heuristic) and must be read as an estimate, not an exact count. Model::is_exact returns false for it.

Heuristic

A tokenizer-agnostic labeled heuristic (no model-specific BPE).

Implementations

impl Model

pub fn name(self) -> &'static str

A human-readable label for the model/tokenizer (e.g. for report output).

Examples found in repository

examples/evaluate.rs (line 37)

fn main() {
    println!("agentic-eval — four-axis evaluation of programs for agentic AI\n");

    // Two encodings of "read a file and keep the large entries".
    let legible = Program::new(
        "legible",
        r#"ls("./src") | where(fn(f) => f.size > 1000) | map(fn(f) => f.name)"#,
    )
    .with_standing_context("ls/where/map are standard, high-probability names")
    .with_output("name\nfoo.rs\nbar.rs");
    let cipher = Program::new("cipher", r#"l./src|w~.size>1k|m~.name"#)
        .with_standing_context("single-letter+sigil cheatsheet line; ".repeat(120))
        .with_output("name\nfoo.rs\nbar.rs")
        .with_retries(8); // terse cipher is mis-emitted more often

    // ── 1. Token efficiency ──────────────────────────────────────────────
    println!("[1] Token efficiency (amortized over 30 turns):");
    for model in [
        Model::OpenAiGpt4,
        Model::OpenAiGpt4o,
        Model::AnthropicClaude,
    ] {
        let cmp = compare(&legible, &cipher, model, 30);
        println!(
            "  {:<28} legible={:>6}  cipher={:>6}  → {} wins ({:.2}x){}",
            model.name(),
            cmp.a_total,
            cmp.b_total,
            if cmp.winner_is_a { "legible" } else { "cipher" },
            cmp.ratio,
            if model.is_exact() { "" } else { " [est]" },
        );
    }

    // ── 2. Determinism ───────────────────────────────────────────────────
    // A canonical renderer (byte-stable) vs. one that embeds a timestamp.
    let canonical = assess_determinism(5, || "name\nfoo.rs\nbar.rs".to_string());
    let mut t = 0u64;
    let noisy = assess_determinism(5, || {
        t += 1;
        format!("name\nfoo.rs\nbar.rs  # at {t}")
    });
    println!("\n[2] Determinism:");
    println!(
        "  canonical output : deterministic={} ({} distinct / {} runs)",
        canonical.deterministic, canonical.distinct, canonical.runs
    );
    println!(
        "  timestamped output: deterministic={} ({} distinct / {} runs)",
        noisy.deterministic, noisy.distinct, noisy.runs
    );

    // ── 3. Reliability ───────────────────────────────────────────────────
    // The legible form parses on all 6 sample invocations; the cipher mis-parses
    // twice but at least returns a structured error once.
    let samples = [0, 1, 2, 3, 4, 5];
    let legible_rel = assess_reliability(&samples, |_| Outcome::ok());
    let cipher_rel = assess_reliability(&samples, |&i| match i {
        4 => Outcome::structured_failure(),
        5 => Outcome::opaque_failure(),
        _ => Outcome::ok(),
    });
    println!("\n[3] Reliability:");
    println!(
        "  legible: pass {:.0}%  actionable {:.0}%",
        legible_rel.pass_rate * 100.0,
        legible_rel.actionable_rate * 100.0
    );
    println!(
        "  cipher : pass {:.0}%  actionable {:.0}%",
        cipher_rel.pass_rate * 100.0,
        cipher_rel.actionable_rate * 100.0
    );

    // ── 4. Safety ────────────────────────────────────────────────────────
    // The task reads + lists (ReadLocal); a "rm the small files" variant adds a
    // Destructive effect. Score the gating under the agent policy.
    let read_only = assess_safety(&[Effect::ReadLocal, Effect::WriteLocal], Mode::Agent);
    let destructive = assess_safety(
        &[Effect::ReadLocal, Effect::Destructive, Effect::Exec],
        Mode::Agent,
    );
    println!("\n[4] Safety (agent policy):");
    println!(
        "  read+write task : grade {} (bounded={}, {} approval-gated)",
        read_only.grade, read_only.bounded, read_only.approval_gated
    );
    println!(
        "  rm+exec task    : grade {} (bounded={}, {} approval-gated, {} denied)",
        destructive.grade, destructive.bounded, destructive.approval_gated, destructive.denied
    );

    println!("\nSummary: the legible form is competitive-or-cheaper on tokens once standing");
    println!("context counts, more deterministic and reliable to parse, and the agent policy");
    println!("bounds the blast radius of even the destructive variant.");
}

pub fn all() -> [Model; 4]

Every model this build can count for (exact or approximate).

pub fn from_name(name: &str) -> Option<Model>

Parse a model from a short identifier (case-insensitive), for CLI/config use. Accepts common aliases: gpt4/gpt-4/cl100k; gpt4o/gpt-4o/ o200k; claude/anthropic; heuristic/heur. Returns None otherwise.

pub fn is_exact(self) -> bool

Whether this model’s count is exact (a real BPE) in this build, vs. an estimate. OpenAI families are exact only with --features real-tokens.

Examples found in repository

examples/tokens_of.rs (line 16)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!(
        "tokenizer exact: cl100k={} o200k={}",
        cl.is_exact(),
        o2.is_exact()
    );
    println!("{:>7} {:>7}   file", "cl100k", "o200k");
    for path in std::env::args().skip(1) {
        match fs::read_to_string(&path) {
            Ok(s) => println!("{:>7} {:>7}   {}", cl.count(&s), o2.count(&s), path),
            Err(e) => println!("    ERR     ERR   {path}: {e}"),
        }
    }
}

examples/vocabulary_audit.rs (line 24)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== Standard-vocabulary tokenizer audit (§8b) ===");
    println!(
        "tokenizer: {}   names: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" },
        VOCAB.len()
    );

    // Agents emit a name with a leading space; BPE is space-aware.
    let mut single = 0usize;
    let mut offenders: Vec<(&str, usize, usize)> = Vec::new();
    for &name in VOCAB {
        let ctx = format!(" {name}");
        let (c, o) = (cl.count(&ctx), o2.count(&ctx));
        if c <= 1 && o <= 1 {
            single += 1;
        } else {
            offenders.push((name, c, o));
        }
    }

    println!("SINGLE BPE TOKEN (both tokenizers): {single}/{}", VOCAB.len());
    if offenders.is_empty() {
        println!("  ✓ every vocabulary name is a single token — the §8b discipline holds.");
    } else {
        println!("\nOFFENDERS (rename or drop — a multi-token name negates the saving):");
        for (n, c, o) in &offenders {
            println!("  {n:<12} cl100k {c}  o200k {o}");
        }
    }

    println!("\nWHY IT MATTERS");
    println!("  The vocabulary's win is naming an intent in ~1 token. A 2-token name (e.g.");
    println!("  `frequencies` = 'frequ'+'encies') halves that. Picking `freq`/`map`/`fold` over");
    println!("  `frequencies`/`transform`/`accumulate` is tokenizer co-design, audited here.");
}

examples/keyword_audit.rs (line 31)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== MechGen keyword tokenizer audit (migration step 4) ===");
    println!(
        "tokenizer: {}   keywords: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" },
        KEYWORDS.len()
    );

    // A keyword usually appears with a leading space in code; BPE is space-aware,
    // so " return" can differ from "return". Audit the in-context form (leading
    // space) — that is what an agent actually emits.
    let mut offenders: Vec<(&str, usize, usize)> = Vec::new();
    let mut single = 0usize;
    for &kw in KEYWORDS {
        let ctx = format!(" {kw}");
        let c = cl.count(&ctx);
        let o = o2.count(&ctx);
        if c <= 1 && o <= 1 {
            single += 1;
        } else {
            offenders.push((kw, c, o));
        }
    }

    println!("SINGLE BPE TOKEN (both tokenizers): {single}/{}", KEYWORDS.len());
    println!("\nOFFENDERS (>1 token in cl100k or o200k):");
    if offenders.is_empty() {
        println!("  (none)");
    } else {
        offenders.sort_by(|a, b| b.1.cmp(&a.1));
        for (kw, c, o) in &offenders {
            println!("  {kw:<20} cl100k {c}  o200k {o}");
        }
    }

    println!("\nVERDICT");
    println!(
        "  {}/{} keywords are already single-token (the agent-mode single/double-char forms",
        single, KEYWORDS.len()
    );
    println!("  f/m/v/u/… and common words if/for/match/… cost exactly one token).");
    if !offenders.is_empty() {
        println!("  The {} offenders are compound/rare words (snake_case splits on `_`); each should", offenders.len());
        println!("  get a single-token agent-mode alias. They are specialized (swarm combinators,");
        println!("  grammar extension) — rare in practice, so the realized token cost is small, but");
        println!("  the surface is not yet uniformly single-token. This is the concrete step-4 work-list.");
    }
}

examples/abstraction_tokens.rs (line 22)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== Abstraction as the post-floor token lever (real BPE) ===");
    println!(
        "tokenizer: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" }
    );

    // (intent, hand-rolled [compiles today], with-vocabulary [proposed primitive])
    let cases: &[(&str, &str, &str)] = &[
        (
            "sum a list",
            "f sum(xs)\n  var t = 0\n  for x in xs\n    t = t + x\n  t",
            "f sum(xs)\n  fold(xs, 0, +)",
        ),
        (
            "word frequencies",
            "f wc(ws)\n  var m = {}\n  for w in ws\n    m[w] = m[w] + 1\n  m",
            "f wc(ws)\n  freq(ws)",
        ),
        (
            "evens, doubled",
            "f f(xs)\n  var out = []\n  for x in xs\n    if x % 2 == 0\n      out.push(x * 2)\n  out",
            "f f(xs)\n  xs | filter even | map double",
        ),
        (
            "max of a list",
            "f max(xs)\n  var m = xs[0]\n  for x in xs\n    if x > m\n      m = x\n  m",
            "f max(xs)\n  reduce(xs, max)",
        ),
    ];

    println!("{:<18} {:>9} {:>9} {:>7}", "intent", "handrolled", "vocab", "saved");
    let (mut h_cl, mut v_cl, mut h_o, mut v_o) = (0, 0, 0, 0);
    for (name, hand, vocab) in cases {
        let (h, v) = (cl.count(hand), cl.count(vocab));
        println!("{name:<18} {h:>9} {v:>9} {:>6}%", 100 - 100 * v / h);
        h_cl += h; v_cl += v;
        h_o += o2.count(hand); v_o += o2.count(vocab);
    }
    println!("\nTOTAL  cl100k {h_cl} → {v_cl} ({}% saved)   o200k {h_o} → {v_o} ({}% saved)",
        100 - 100 * v_cl / h_cl, 100 - 100 * v_o / h_o);

    println!("\nFINDING");
    println!("  At the surface floor, abstraction is the only per-call token lever left, and it is");
    println!("  POSITIVE-SUM: a single-token, total, capability-typed primitive (a) cuts payload");
    println!("  tokens (above), (b) RAISES reliability (no hand-rolled off-by-one / empty-list bug),");
    println!("  and (c) preserves safety (the primitive's effect rides its type to the boundary).");
    println!("  Encoding tricks (binary, dense UTF-8) and layout were all token-neutral-or-worse —");
    println!("  vocabulary is the one that pays. The discipline: name primitives as single BPE tokens,");
    println!("  make them total, and choose them by the empirical frequency of SWE intents.");
}

examples/rain_tokens.rs (line 88)

fn main() {
    let exact = Model::OpenAiGpt4.is_exact();
    println!("=== MechGen 'digital rain' vs token streams (cl100k + o200k BPE) ===");
    println!(
        "tokenizer: {}\n",
        if exact { "REAL tiktoken (exact)" } else { "HEURISTIC (CJK undercounted — rerun with --features real-tokens)" }
    );

    let samples: &[(&str, &str)] = &[
        ("net", "net MLP { layer fc1: Linear(8, 16); layer act: ReLU; layer fc2: Linear(16, 4); forward { fc1 } }"),
        ("fn", "fn factorial(n: u64) -> u64 { if n <= 1 { return 1; } n * factorial(n - 1) }"),
        ("kb", "kb Family { fact parent(alice, bob); fact parent(bob, carol); rule gp(x: i32, z: i32) where parent(x, y), parent(y, z) { x } }"),
    ];

    println!("{:<5} {:>10} {:>22} {:>22} {:>14}", "kind", "form", "cl100k tok", "o200k tok", "chars");
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    let (mut s_cl, mut s_o, mut r_cl, mut r_o, mut rl_cl, mut rl_o) = (0, 0, 0, 0, 0, 0);
    for (name, src) in samples {
        let (stream, legend) = rain(src);
        let rl = format!("{legend}\n{stream}"); // rain + legend (one-off, reversible)
        let bytes = src.as_bytes();
        let base = b64(bytes);
        let row = |label: &str, s: &str| {
            println!("{:<5} {:>10} {:>22} {:>22} {:>14}", "", label, cl.count(s), o2.count(s), chars(s));
        };
        println!("[{name}]");
        row("source", src);
        row("rain(stream)", &stream);
        row("rain+legend", &rl);
        row("base64(bytes)", &base);
        s_cl += cl.count(src); s_o += o2.count(src);
        r_cl += cl.count(&stream); r_o += o2.count(&stream);
        rl_cl += cl.count(&rl); rl_o += o2.count(&rl);
    }

    println!("\nTOTALS (3 samples) — token ratio vs source (>1.0 = rain is WORSE):");
    println!("  source            cl100k {s_cl:>4}   o200k {s_o:>4}");
    println!("  rain(stream)      cl100k {r_cl:>4} ({:.2}x)   o200k {r_o:>4} ({:.2}x)", r_cl as f64 / s_cl as f64, r_o as f64 / s_o as f64);
    println!("  rain+legend       cl100k {rl_cl:>4} ({:.2}x)   o200k {rl_o:>4} ({:.2}x)", rl_cl as f64 / s_cl as f64, rl_o as f64 / s_o as f64);

    println!("\nVERDICT");
    println!("  Digital rain shrinks CHARACTERS (~3x — the Matrix look) but the dense glyph");
    println!("  stream costs MORE BPE tokens than the ASCII source (ratios above), because the");
    println!("  tokenizer splits each rare multi-byte glyph into several tokens. Adding the legend");
    println!("  (needed for reversibility) makes a one-off snippet far worse still. Even an");
    println!("  amortized shared codebook (stream only) does not beat source on tokens.");
    println!("  This is the project's token-floor finding, re-confirmed on the dense-symbol idea:");
    println!("  an LLM emits TOKENS, not glyphs/bytes — the information (names/ops/dims) is the floor.");
    if !exact {
        println!("\n  (heuristic run: it counts ~1 token/char and UNDER-counts CJK — the real cl100k/o200k");
        println!("   gap is larger. Rerun with --features real-tokens for the exact, even-worse numbers.)");
    }
}

examples/design_tokens.rs (line 24)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== Token-efficiency design levers (real cl100k + o200k BPE) ===");
    println!(
        "tokenizer: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" }
    );

    // Each task: (name, ceremony-heavy, current-ish, ab-initio).
    let tasks: &[(&str, &str, &str, &str)] = &[
        (
            "word-count",
            // A — ceremony-heavy
            "use std::collections::HashMap;\n\nfn count_words(text: &str) -> HashMap<String, u32> {\n    let mut counts: HashMap<String, u32> = HashMap::new();\n    for word in text.split_whitespace() {\n        *counts.entry(word.to_string()).or_insert(0) += 1;\n    }\n    counts\n}",
            // B — current-MechGen-ish (sigils, var, some inference)
            "fn count_words(text: &str) -> {s: u32} {\n    var counts = {s: u32}.new()\n    for word in text.split() {\n        counts.entry(word).or(0) += 1\n    }\n    counts\n}",
            // C — ab-initio (inference + layout + ambient builtins)
            "count_words text =\n  counts = {}\n  for w in split text\n    counts[w] += 1\n  counts",
        ),
        (
            "factorial",
            "fn factorial(n: u64) -> u64 {\n    if n <= 1 {\n        return 1;\n    }\n    n * factorial(n - 1)\n}",
            "fn factorial(n: u64) -> u64 {\n    if n <= 1 { 1 } else { n * factorial(n - 1) }\n}",
            "fact n =\n  if n <= 1: 1\n  else: n * fact (n - 1)",
        ),
        (
            "safe-divide", // returns optional/result — safety ceremony vs sigil
            "fn safe_div(a: i32, b: i32) -> Option<i32> {\n    if b == 0 {\n        return None;\n    }\n    Some(a / b)\n}",
            "fn safe_div(a: i32, b: i32) -> ?i32 {\n    if b == 0 { none } else { a / b }\n}",
            "div a b =\n  if b == 0: none\n  else: a / b",
        ),
    ];

    println!("{:<13} {:>4} {:>9} {:>8} {:>7}", "task", "form", "cl100k", "o200k", "chars");
    let (mut a_cl, mut b_cl, mut c_cl) = (0, 0, 0);
    let (mut a_o, mut b_o, mut c_o) = (0, 0, 0);
    for (name, a, b, c) in tasks {
        let row = |label: &str, s: &str| {
            println!("{:<13} {:>4} {:>9} {:>8} {:>7}", "", label, cl.count(s), o2.count(s), s.chars().count());
        };
        println!("[{name}]");
        row("A heavy", a);
        row("B curr", b);
        row("C abinit", c);
        a_cl += cl.count(a); b_cl += cl.count(b); c_cl += cl.count(c);
        a_o += o2.count(a); b_o += o2.count(b); c_o += o2.count(c);
    }

    println!("\nTOTALS (3 tasks):");
    println!("  A ceremony-heavy   cl100k {a_cl:>3}   o200k {a_o:>3}   (baseline)");
    println!("  B current-ish      cl100k {b_cl:>3} ({:.0}%)   o200k {b_o:>3} ({:.0}%)", 100.0 * b_cl as f64 / a_cl as f64, 100.0 * b_o as f64 / a_o as f64);
    println!("  C ab-initio        cl100k {c_cl:>3} ({:.0}%)   o200k {c_o:>3} ({:.0}%)", 100.0 * c_cl as f64 / a_cl as f64, 100.0 * c_o as f64 / a_o as f64);
    println!("\n  → ab-initio cuts ~{:.0}% of cl100k tokens vs ceremony-heavy by REMOVING ceremony",
        100.0 * (1.0 - c_cl as f64 / a_cl as f64));
    println!("    (types/mutability/return/imports inferred; layout replaces braces+`;`; terse safety");
    println!("    sigils; ambient builtins). The remaining tokens are the irreducible payload —");
    println!("    names/ops/literals — which no design can remove. That residue IS the token floor.");
}

Additional examples can be found in:

• examples/inference_tokens.rs
• examples/evaluate.rs
• examples/swe_token_benchmark.rs

pub fn count(self, text: &str) -> usize

Count the tokens in text under this model.

Examples found in repository

examples/tokens_of.rs (line 22)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!(
        "tokenizer exact: cl100k={} o200k={}",
        cl.is_exact(),
        o2.is_exact()
    );
    println!("{:>7} {:>7}   file", "cl100k", "o200k");
    for path in std::env::args().skip(1) {
        match fs::read_to_string(&path) {
            Ok(s) => println!("{:>7} {:>7}   {}", cl.count(&s), o2.count(&s), path),
            Err(e) => println!("    ERR     ERR   {path}: {e}"),
        }
    }
}

examples/vocabulary_audit.rs (line 33)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== Standard-vocabulary tokenizer audit (§8b) ===");
    println!(
        "tokenizer: {}   names: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" },
        VOCAB.len()
    );

    // Agents emit a name with a leading space; BPE is space-aware.
    let mut single = 0usize;
    let mut offenders: Vec<(&str, usize, usize)> = Vec::new();
    for &name in VOCAB {
        let ctx = format!(" {name}");
        let (c, o) = (cl.count(&ctx), o2.count(&ctx));
        if c <= 1 && o <= 1 {
            single += 1;
        } else {
            offenders.push((name, c, o));
        }
    }

    println!("SINGLE BPE TOKEN (both tokenizers): {single}/{}", VOCAB.len());
    if offenders.is_empty() {
        println!("  ✓ every vocabulary name is a single token — the §8b discipline holds.");
    } else {
        println!("\nOFFENDERS (rename or drop — a multi-token name negates the saving):");
        for (n, c, o) in &offenders {
            println!("  {n:<12} cl100k {c}  o200k {o}");
        }
    }

    println!("\nWHY IT MATTERS");
    println!("  The vocabulary's win is naming an intent in ~1 token. A 2-token name (e.g.");
    println!("  `frequencies` = 'frequ'+'encies') halves that. Picking `freq`/`map`/`fold` over");
    println!("  `frequencies`/`transform`/`accumulate` is tokenizer co-design, audited here.");
}

examples/keyword_audit.rs (line 42)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== MechGen keyword tokenizer audit (migration step 4) ===");
    println!(
        "tokenizer: {}   keywords: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" },
        KEYWORDS.len()
    );

    // A keyword usually appears with a leading space in code; BPE is space-aware,
    // so " return" can differ from "return". Audit the in-context form (leading
    // space) — that is what an agent actually emits.
    let mut offenders: Vec<(&str, usize, usize)> = Vec::new();
    let mut single = 0usize;
    for &kw in KEYWORDS {
        let ctx = format!(" {kw}");
        let c = cl.count(&ctx);
        let o = o2.count(&ctx);
        if c <= 1 && o <= 1 {
            single += 1;
        } else {
            offenders.push((kw, c, o));
        }
    }

    println!("SINGLE BPE TOKEN (both tokenizers): {single}/{}", KEYWORDS.len());
    println!("\nOFFENDERS (>1 token in cl100k or o200k):");
    if offenders.is_empty() {
        println!("  (none)");
    } else {
        offenders.sort_by(|a, b| b.1.cmp(&a.1));
        for (kw, c, o) in &offenders {
            println!("  {kw:<20} cl100k {c}  o200k {o}");
        }
    }

    println!("\nVERDICT");
    println!(
        "  {}/{} keywords are already single-token (the agent-mode single/double-char forms",
        single, KEYWORDS.len()
    );
    println!("  f/m/v/u/… and common words if/for/match/… cost exactly one token).");
    if !offenders.is_empty() {
        println!("  The {} offenders are compound/rare words (snake_case splits on `_`); each should", offenders.len());
        println!("  get a single-token agent-mode alias. They are specialized (swarm combinators,");
        println!("  grammar extension) — rare in practice, so the realized token cost is small, but");
        println!("  the surface is not yet uniformly single-token. This is the concrete step-4 work-list.");
    }
}

examples/abstraction_tokens.rs (line 52)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== Abstraction as the post-floor token lever (real BPE) ===");
    println!(
        "tokenizer: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" }
    );

    // (intent, hand-rolled [compiles today], with-vocabulary [proposed primitive])
    let cases: &[(&str, &str, &str)] = &[
        (
            "sum a list",
            "f sum(xs)\n  var t = 0\n  for x in xs\n    t = t + x\n  t",
            "f sum(xs)\n  fold(xs, 0, +)",
        ),
        (
            "word frequencies",
            "f wc(ws)\n  var m = {}\n  for w in ws\n    m[w] = m[w] + 1\n  m",
            "f wc(ws)\n  freq(ws)",
        ),
        (
            "evens, doubled",
            "f f(xs)\n  var out = []\n  for x in xs\n    if x % 2 == 0\n      out.push(x * 2)\n  out",
            "f f(xs)\n  xs | filter even | map double",
        ),
        (
            "max of a list",
            "f max(xs)\n  var m = xs[0]\n  for x in xs\n    if x > m\n      m = x\n  m",
            "f max(xs)\n  reduce(xs, max)",
        ),
    ];

    println!("{:<18} {:>9} {:>9} {:>7}", "intent", "handrolled", "vocab", "saved");
    let (mut h_cl, mut v_cl, mut h_o, mut v_o) = (0, 0, 0, 0);
    for (name, hand, vocab) in cases {
        let (h, v) = (cl.count(hand), cl.count(vocab));
        println!("{name:<18} {h:>9} {v:>9} {:>6}%", 100 - 100 * v / h);
        h_cl += h; v_cl += v;
        h_o += o2.count(hand); v_o += o2.count(vocab);
    }
    println!("\nTOTAL  cl100k {h_cl} → {v_cl} ({}% saved)   o200k {h_o} → {v_o} ({}% saved)",
        100 - 100 * v_cl / h_cl, 100 - 100 * v_o / h_o);

    println!("\nFINDING");
    println!("  At the surface floor, abstraction is the only per-call token lever left, and it is");
    println!("  POSITIVE-SUM: a single-token, total, capability-typed primitive (a) cuts payload");
    println!("  tokens (above), (b) RAISES reliability (no hand-rolled off-by-one / empty-list bug),");
    println!("  and (c) preserves safety (the primitive's effect rides its type to the boundary).");
    println!("  Encoding tricks (binary, dense UTF-8) and layout were all token-neutral-or-worse —");
    println!("  vocabulary is the one that pays. The discipline: name primitives as single BPE tokens,");
    println!("  make them total, and choose them by the empirical frequency of SWE intents.");
}

examples/rain_tokens.rs (line 111)

fn main() {
    let exact = Model::OpenAiGpt4.is_exact();
    println!("=== MechGen 'digital rain' vs token streams (cl100k + o200k BPE) ===");
    println!(
        "tokenizer: {}\n",
        if exact { "REAL tiktoken (exact)" } else { "HEURISTIC (CJK undercounted — rerun with --features real-tokens)" }
    );

    let samples: &[(&str, &str)] = &[
        ("net", "net MLP { layer fc1: Linear(8, 16); layer act: ReLU; layer fc2: Linear(16, 4); forward { fc1 } }"),
        ("fn", "fn factorial(n: u64) -> u64 { if n <= 1 { return 1; } n * factorial(n - 1) }"),
        ("kb", "kb Family { fact parent(alice, bob); fact parent(bob, carol); rule gp(x: i32, z: i32) where parent(x, y), parent(y, z) { x } }"),
    ];

    println!("{:<5} {:>10} {:>22} {:>22} {:>14}", "kind", "form", "cl100k tok", "o200k tok", "chars");
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    let (mut s_cl, mut s_o, mut r_cl, mut r_o, mut rl_cl, mut rl_o) = (0, 0, 0, 0, 0, 0);
    for (name, src) in samples {
        let (stream, legend) = rain(src);
        let rl = format!("{legend}\n{stream}"); // rain + legend (one-off, reversible)
        let bytes = src.as_bytes();
        let base = b64(bytes);
        let row = |label: &str, s: &str| {
            println!("{:<5} {:>10} {:>22} {:>22} {:>14}", "", label, cl.count(s), o2.count(s), chars(s));
        };
        println!("[{name}]");
        row("source", src);
        row("rain(stream)", &stream);
        row("rain+legend", &rl);
        row("base64(bytes)", &base);
        s_cl += cl.count(src); s_o += o2.count(src);
        r_cl += cl.count(&stream); r_o += o2.count(&stream);
        rl_cl += cl.count(&rl); rl_o += o2.count(&rl);
    }

    println!("\nTOTALS (3 samples) — token ratio vs source (>1.0 = rain is WORSE):");
    println!("  source            cl100k {s_cl:>4}   o200k {s_o:>4}");
    println!("  rain(stream)      cl100k {r_cl:>4} ({:.2}x)   o200k {r_o:>4} ({:.2}x)", r_cl as f64 / s_cl as f64, r_o as f64 / s_o as f64);
    println!("  rain+legend       cl100k {rl_cl:>4} ({:.2}x)   o200k {rl_o:>4} ({:.2}x)", rl_cl as f64 / s_cl as f64, rl_o as f64 / s_o as f64);

    println!("\nVERDICT");
    println!("  Digital rain shrinks CHARACTERS (~3x — the Matrix look) but the dense glyph");
    println!("  stream costs MORE BPE tokens than the ASCII source (ratios above), because the");
    println!("  tokenizer splits each rare multi-byte glyph into several tokens. Adding the legend");
    println!("  (needed for reversibility) makes a one-off snippet far worse still. Even an");
    println!("  amortized shared codebook (stream only) does not beat source on tokens.");
    println!("  This is the project's token-floor finding, re-confirmed on the dense-symbol idea:");
    println!("  an LLM emits TOKENS, not glyphs/bytes — the information (names/ops/dims) is the floor.");
    if !exact {
        println!("\n  (heuristic run: it counts ~1 token/char and UNDER-counts CJK — the real cl100k/o200k");
        println!("   gap is larger. Rerun with --features real-tokens for the exact, even-worse numbers.)");
    }
}

examples/design_tokens.rs (line 57)

fn main() {
    let cl = Model::OpenAiGpt4;
    let o2 = Model::OpenAiGpt4o;
    println!("=== Token-efficiency design levers (real cl100k + o200k BPE) ===");
    println!(
        "tokenizer: {}\n",
        if cl.is_exact() { "REAL tiktoken (exact)" } else { "HEURISTIC — rerun with --features real-tokens" }
    );

    // Each task: (name, ceremony-heavy, current-ish, ab-initio).
    let tasks: &[(&str, &str, &str, &str)] = &[
        (
            "word-count",
            // A — ceremony-heavy
            "use std::collections::HashMap;\n\nfn count_words(text: &str) -> HashMap<String, u32> {\n    let mut counts: HashMap<String, u32> = HashMap::new();\n    for word in text.split_whitespace() {\n        *counts.entry(word.to_string()).or_insert(0) += 1;\n    }\n    counts\n}",
            // B — current-MechGen-ish (sigils, var, some inference)
            "fn count_words(text: &str) -> {s: u32} {\n    var counts = {s: u32}.new()\n    for word in text.split() {\n        counts.entry(word).or(0) += 1\n    }\n    counts\n}",
            // C — ab-initio (inference + layout + ambient builtins)
            "count_words text =\n  counts = {}\n  for w in split text\n    counts[w] += 1\n  counts",
        ),
        (
            "factorial",
            "fn factorial(n: u64) -> u64 {\n    if n <= 1 {\n        return 1;\n    }\n    n * factorial(n - 1)\n}",
            "fn factorial(n: u64) -> u64 {\n    if n <= 1 { 1 } else { n * factorial(n - 1) }\n}",
            "fact n =\n  if n <= 1: 1\n  else: n * fact (n - 1)",
        ),
        (
            "safe-divide", // returns optional/result — safety ceremony vs sigil
            "fn safe_div(a: i32, b: i32) -> Option<i32> {\n    if b == 0 {\n        return None;\n    }\n    Some(a / b)\n}",
            "fn safe_div(a: i32, b: i32) -> ?i32 {\n    if b == 0 { none } else { a / b }\n}",
            "div a b =\n  if b == 0: none\n  else: a / b",
        ),
    ];

    println!("{:<13} {:>4} {:>9} {:>8} {:>7}", "task", "form", "cl100k", "o200k", "chars");
    let (mut a_cl, mut b_cl, mut c_cl) = (0, 0, 0);
    let (mut a_o, mut b_o, mut c_o) = (0, 0, 0);
    for (name, a, b, c) in tasks {
        let row = |label: &str, s: &str| {
            println!("{:<13} {:>4} {:>9} {:>8} {:>7}", "", label, cl.count(s), o2.count(s), s.chars().count());
        };
        println!("[{name}]");
        row("A heavy", a);
        row("B curr", b);
        row("C abinit", c);
        a_cl += cl.count(a); b_cl += cl.count(b); c_cl += cl.count(c);
        a_o += o2.count(a); b_o += o2.count(b); c_o += o2.count(c);
    }

    println!("\nTOTALS (3 tasks):");
    println!("  A ceremony-heavy   cl100k {a_cl:>3}   o200k {a_o:>3}   (baseline)");
    println!("  B current-ish      cl100k {b_cl:>3} ({:.0}%)   o200k {b_o:>3} ({:.0}%)", 100.0 * b_cl as f64 / a_cl as f64, 100.0 * b_o as f64 / a_o as f64);
    println!("  C ab-initio        cl100k {c_cl:>3} ({:.0}%)   o200k {c_o:>3} ({:.0}%)", 100.0 * c_cl as f64 / a_cl as f64, 100.0 * c_o as f64 / a_o as f64);
    println!("\n  → ab-initio cuts ~{:.0}% of cl100k tokens vs ceremony-heavy by REMOVING ceremony",
        100.0 * (1.0 - c_cl as f64 / a_cl as f64));
    println!("    (types/mutability/return/imports inferred; layout replaces braces+`;`; terse safety");
    println!("    sigils; ambient builtins). The remaining tokens are the irreducible payload —");
    println!("    names/ops/literals — which no design can remove. That residue IS the token floor.");
}

Additional examples can be found in:

• examples/inference_tokens.rs
• examples/swe_token_benchmark.rs

Trait Implementations

impl Clone for Model

fn clone(&self) -> Model

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Model

impl Debug for Model

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Model

impl PartialEq for Model

fn eq(&self, other: &Model) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Model

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Model