mongol-norm

English | 中文

---

English

mongol-norm is a shape-aware normalizer for Traditional Mongolian (Hudum) script: the full UTN #57 v4 shaping pipeline plus an FVS-pinned canonical normalizer, in a Rust crate with zero dependencies that also builds for wasm32-unknown-unknown.

The same visible Mongolian word can be written as many different Unicode sequences, which breaks search, deduplication and indexing. shape() renders a word into its written-unit sequence — what a font would draw, computed without a font — and normalize() rewrites any encoding of a word into the one canonical encoding for that shape.

The crate is the engine and lives at the repository root. python/ holds a thin PyO3 binding, published to PyPI as mongol-norm; both are built from one version literal, one set of data tables and one fixture set, and return byte-identical results — see Python package.

Status

Both halves of the library — shaping and normalization (MNG / Hudum) — are verified against the same upstream corpora on every push, on Rust stable and the MSRV, and on CPython 3.9 – 3.14.

✅ Shaping

shape() and same_shape() are cross-validated against two upstream TSV suites:

Suite	Cases	Pass	Notes
mongfontbuilder/core-hud.tsv	225	100%	curated regression set
mongfontbuilder/eac-hud.tsv (GB/T 25914-2023)	3513	100%	5 cases excluded as UTN ↔ EAC xfail, matching mongfontbuilder's own pytest.mark.xfail set
Hand-written unit tests	—	100%	shape / same_shape / joiner tokens (nirugu, ZWJ)

Both TSV suites are the standard's own written-unit sequences, so they are checked against shape_raw — the engine's output before the nine duplicate encodings are unified (see Duplicate encodings). For 375 of the 3512 EAC rows and 30 of the 177 core rows the public shape differs from what the standard spells.

✅ Normalization — corpus-scoped guarantees, machine-checked

For written-unit shapes covered by the bundled normalization table, normalize / normalize_text / normalize_written_units are pure functions of shape, with invariants checked in CI over every corpus encoding:

Property	Result
Round-trip — shape(normalize(x)) == shape(x)	3757 / 3757 corpus encodings (100%)
Shape-canonicity — same shape ⟹ same Unicode output	1990 / 1990 shape groups (100%)
Prefix-stability — word and word+suffix share their prefix encoding	2240 / 2240 real corpus pairs (100%)

Scope note: normalization is implemented for MNG (Hudum) only — Todo / Sibe / Manchu load shaping rules but have no normalizer yet. The guarantees above cover the checked corpus and any input whose written-unit chains can be encoded by the bundled table. For an uncovered out-of-corpus chain, normalization fails closed with Error::NormalizationFallback; call normalize_allow_fallback explicitly when returning the original word unchanged is acceptable.

This project was generated with Claude Code (AI-assisted coding). The tests and key parts of the core code have been manually reviewed, and test coverage is extensive (corpus round-trip / shape-canonicity / prefix-stability plus the upstream cross-implementation suites). Treat this as a preview release — it should be fine for normal use; if you hit a problem, please open an issue or PR. Shaping logic is derived from UTN #57 v4 and mongfontbuilder.

Install

cargo add mongol-norm

or, by hand:

[dependencies]
mongol-norm = "0.2.0"

The CLI installs as a standalone binary:

cargo install mongol-norm

MSRV is 1.82. API docs: docs.rs/mongol-norm.

Usage

use mongol_norm::{Error, Locale, PositionedWrittenUnit, Shaper, UnitPosition, WrittenUnit};

fn main() -> Result<(), Error> {
    let shaper = Shaper::new(Locale::Mng);

    // Shape: written-unit sequence
    let shape = shaper.shape("ᠰᠠᠢᠨ")?;
    assert_eq!(shape.len(), 5); // [S, A, I, I, A]
    assert_eq!(shaper.shape_str("ᠰᠠᠢᠨ")?, "S+A+I+I+A");

    // Compare: visually identical?
    assert!(shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")?);

    // Normalize a word to its canonical, FVS-pinned encoding (strict: an uncovered shape is an
    // error — `normalize_allow_fallback` returns such input unchanged instead)
    let canonical = shaper.normalize("ᠰᠡᠢᠨ")?;
    assert_eq!(shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")?, canonical);

    // Free-form text: Mongolian words normalized, everything else preserved
    let text = shaper.normalize_text("Hello ᠰᠡᠢᠨ world")?;
    assert!(text.starts_with("Hello ") && text.ends_with(" world"));

    // Written units and authoritative HUD positions
    let units = [WrittenUnit::B, WrittenUnit::Aa];
    assert_eq!(shaper.normalize_written_units(&units)?, "ᠪᠠ᠋");
    let records = [
        PositionedWrittenUnit::new(WrittenUnit::B, UnitPosition::Init),
        PositionedWrittenUnit::new(WrittenUnit::Aa, UnitPosition::Fina),
    ];
    assert_eq!(shaper.normalize_positioned_written_units(&records)?, "ᠪᠠ᠋");
    assert_eq!(shaper.canonical_version(), Some("mng-canonical/2"));

    // The public shape unifies the nine duplicate encodings, so ᠠᠷᠠᠳ and ᠠᠷᠠᠤᠠ — one visible
    // word spelled two ways — are one shape and one canonical text
    assert_eq!(shaper.shape_str("ᠠᠷᠠᠳ")?, "A+A+R+A+O+A");
    assert!(shaper.same_shape("ᠠᠷᠠᠳ", "ᠠᠷᠠᠤᠠ")?);
    assert_eq!(shaper.normalize("ᠠᠷᠠᠳ")?, shaper.normalize("ᠠᠷᠠᠤᠠ")?);
    Ok(())
}

shape_detailed returns the per-token breakdown (code point, alias, position, FVS, condition, written units) and trace returns the per-rule condition transitions behind the phase-trace golden fixtures. The full API is on docs.rs.

Command line

The crate ships a mongol-norm binary — the same CLI the Python package installs as its mongol-norm command. It turns a word into its rendered written-unit sequence (shape), rewrites any encoding of a word into the one canonical, FVS-pinned Unicode form (normalize), does the same for every Mongolian word inside free-form text (normalize-text), encodes pre-shaped written units (normalize-written-units), and tells whether two encodings render identically (same):

cargo install mongol-norm

# Inline text
mongol-norm shape 'ᠰᠠᠢᠨ'                        # → S+A+I+I+A (the rendered written units)
mongol-norm normalize 'ᠰᠡᠢᠨ'                    # one word → its canonical encoding
mongol-norm normalize --allow-fallback 'ᠰᠡᠢᠨ'   # keep the input if it is uncovered
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'         # mixed script: Mongolian runs normalized
mongol-norm normalize-written-units 'B+Aa'      # → ᠪᠠ᠋
mongol-norm normalize-written-units 'BZwj'      # compact PascalCase units
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'                  # exit 0 if identical, 1 if different

# Pipe / stdin (use `-` as the text)
echo 'B+Aa' | mongol-norm normalize-written-units -
cat doc.txt | mongol-norm normalize-text -

# File in / out
mongol-norm normalize-text -i in.txt -o out.txt

# Batch: one word per line in, one canonical per line out
mongol-norm normalize --batch -i words.txt -o canonical.txt
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize --batch -
mongol-norm normalize-written-units --batch -i units.txt -o canonical.txt

--locale MNG|TOD|SIB|MCH selects the script (default MNG; only MNG normalizes). It is a global option and goes before the sub-command — mongol-norm --locale TOD shape 'ᠰᠠᠢᠨ', not after it. --allow-fallback keeps an uncovered word instead of failing, -- ends the options so a following - is text, errors exit with code 2 (same prints true/false and exits 0/1), and mongol-norm --help lists every flag.

shape and normalize take a single word and reject every character outside the Mongolian word alphabet — including the newline echo appends, so pipe them with printf, with --batch (one word per line), or use normalize-text for free-form text.

normalize-written-units accepts compact PascalCase or explicit + boundaries. Compact input must have one unique segmentation; ambiguous input fails closed and must be rewritten with +. After parsing, the same exact-shape validation as the normalize_written_units API applies, so a syntactically valid unit stream can still be rejected when it has no canonical MNG encoding.

Python package

pip install mongol-norm

The PyPI package is a thin PyO3 wrapper over this crate, built by maturin from python/. The engine and its tables are compiled into the mongol_norm._native extension, so there are no runtime dependencies and no Rust toolchain to install.

Prebuilt cp39-abi3 wheels (one per platform, serving every CPython ≥ 3.9) cover Linux x86_64 and aarch64 (glibc via manylinux2014, musl via musllinux_1_2), macOS x86_64 and Apple silicon, and Windows x64; CI tests CPython 3.9 – 3.14. On any other platform pip falls back to the source distribution, which compiles the extension locally and needs a Rust toolchain ≥ 1.83 (pip fetches the maturin build backend itself).

from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")

shaper.shape("ᠰᠠᠢᠨ")                        # → ['S', 'A', 'I', 'I', 'A']
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")           # → True
shaper.normalize("ᠰᠡᠢᠨ")                    # → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")   # → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'
shaper.normalize_written_units(["B", "Aa"]) # → 'ᠪᠠ᠋'

The mongol-norm console script is the crate's CLI, run in-process. Full Python documentation — the whole API, error types, the written-unit encoders and the CLI — is in python/README.pypi.md, which is also the package's PyPI page.

Upgrading from 0.0.x. The public API (MongolianShaper, NormalizationFallbackError, the mongol-norm command) is unchanged; from mongol_norm.shaper import … and python -m mongol_norm.shaper … still work through a compatibility shim. Differences: Python ≥ 3.9 is required; an unknown locale raises ValueError instead of FileNotFoundError; mongol_norm.rules and the private shaper internals no longer exist; the CLI is the crate's (adds -V/--version; its remaining intentional differences are listed at the top of src/cli.rs). New: shaper.trace(), shaper.rule_names(), shaper.parse_written_units().

Why this project exists

Traditional Mongolian script in Unicode has a fundamental problem: the same visible word can be encoded in multiple different Unicode sequences. This happens because:

1. Letters share glyphs — A and E look identical in medial and final positions; O/U and OE/UE share forms; QA and GA share forms depending on vowel harmony.
2. Multiple encoding paths — The same tooth glyph (I) can be encoded as I, YA+FVS1, or even two separate I characters.
3. Redundant FVS usage — Free Variation Selectors (FVS1–FVS4) can create equivalent sequences that render identically.
4. Joining controls — nirugu (U+180A, the visible stem extender) and ZWJ (U+200D) force letters into their joined forms, and inside those joined contexts even more letters collapse to the same glyph (nirugu+o and nirugu+u render identically).
5. Suffix particles after MVS/NNBSP — the same rendered suffix can be spelled with different letters (MVS+a and MVS+e both render the chachlag form; MVS+u and MVS+ue render the same connector).

This means:

• Search fails: Searching for "sain" (one encoding) won't find the same word in another encoding, even though they look identical.
• Deduplication breaks: The same word has multiple Unicode representations.
• Indexing is unreliable: Different encodings of the same word produce different keys.

mongol-norm shapes the input with the full UTN #57 v4 shaping process (5-step conditional mapping), compares glyph sequences to detect identical visual forms, and normalizes supported shapes to one canonical, FVS-pinned Unicode encoding — same shape ⟹ same Unicode, with an exact shape round-trip.

Example: All five of these encode the word "sain" (good) and look identical:

How it works

The normalizer implements a lightweight Mongolian shaping engine — equivalent to what HarfBuzz does with a font file, but using only the rule data from UTN #57 v4 and the mongfontbuilder project. No font files needed.

Shaping pipeline (UTN #57 v4 Mongolian-specific phase)

1. Chachlag — Suffix forms for A/E after MVS (Mongolian Vowel Separator)
2. Syllabic — Consonant/vowel context: onset, devsger, marked, masculine/feminine harmony, dotless
3. Particle — MVS particle dictionary lookup for specific suffix words
4. Devsger — I after a vowel (vowel_devsger) gets double-tooth form: I → I+I
5. Post-bowed — Vowel forms change after bowed consonants (G, B, K, P, F)

Duplicate encodings

shape promises to be a fingerprint of the visible word — "same shape ⟹ same normalize" is this crate's whole reason to exist. Nine written units break that promise: they render as exactly the same ink as a sequence of other units, so two spellings of one word shaped differently.

Five are unified by expanding the single unit into the pair:

duplicate	public shape	witness pair
Dd:medi	O:medi A:medi	ᠣᠳᠪᠣ / ᠠ᠋ᠣᠣᠠᠪᠣ᠋
Dd:fina	O:medi A:fina	ᠠᠷᠠᠳ / ᠠᠷᠠᠤᠠ
H:medi	A:medi A:medi	ᠪᠠᠭᠰᠢ / ᠪᠠᠠᠠᠰᠢ
Hx:medi	N:medi N:medi	ᠠᠷᠭᠠᠯ / ᠠ᠋ᠠᠷᠨ᠋ᠨ᠋ᠠᠯ
Cr:init	O:init O:medi	ᡂ᠊ / ᠤ᠋ᠤ᠊

Four other forms unify by contraction, choosing the shorter canonical form only in the verified position and context. Aa:fina has a tooth immediately after a bowed written unit and no tooth elsewhere; repeated insertion/deletion of A would not preserve the ink:

pair	public shape	witness pair
A Aa spanning a whole chain	A:isol	ᠡ / ᠡᠠ᠋
bowed written unit A:medi Aa:fina	bowed written unit Aa:fina	ᠪᠠ / ᠪᠠᠠ᠋
O Aa ending a chain	B2:fina	᠊ᠪ᠋ / ᠊ᠤᠠ᠋
I Aa ending a chain	G:fina	᠊ᠭ / ᠊ᠢᠠ᠋

Position is part of every rule, and it is the chain position normalize already uses — slots between structural units, with a nirugu or ZWJ neighbour padding the chain the way it pads the rendering. That is why the B2 and G witnesses are written with a leading nirugu: it is what makes the unit final. Forms outside a verified pair are left alone — initial and final H/Hx are distinct ink, a lone Cr chain is Cr:isol rather than the verified Cr:init, and a lone A chain is already canonical.

Terminology. UTN #57 uses bowed written units (圆头书写单位) and names the corresponding lookup Post-bowed; see UTN #57 revision 4.

Context. The complete Hudum bowed written unit set is B, P, F, G, Gx, K, K2, not every consonant. Sources: the Hudum written-unit ligated variants, required ligatures (BAa, PAa, FAa, GAa, GxAa, KAa, K2Aa), and src/rules.rs's post-bowed classes. The bowed written unit must immediately precede the medial A in the same chain: B A Aa → B Aa, but N A Aa, A A Aa, and B A A Aa stay unchanged. Joiner padding is not a bowed written unit and does not let this rule cross structural tokens. The independent whole-chain A:init Aa:fina → A:isol rule is unchanged.

Termination and idempotence. Expand once, then inspect the final pair once. Expansion emits no expansion targets, and contraction cannot expose an initial/final H/Hx as medial. Every contraction ends the chain: A, B2, and G do not end in Aa; a contracted Aa is preceded by a bowed written unit, not A/O/I, so it cannot contract again. Idempotence follows from these guards, not from forcing the text to a fixed point. The 20-unit alphabet (all bowed written units, expansion targets and structural tokens included) is exhausted through length four: 168 421 inputs. Corpus written-unit re-encoding also reshapes to itself.

Expansion does not license a non-bowed written unit contraction: ᠲᠡᠳ᠌ᠡ᠋ (T A Dd Aa) becomes T A O A Aa, not T A B2, because O is not a bowed written unit. The expansion of B H Aa likewise leaves B A A Aa. Both interactions are regression-tested.

Everything user-facing sees the unified sequence: same_shape, normalize and the written-unit encoders. normalize_written_units still accepts the duplicates as input and unifies them, so existing caller data keeps working. ᠠᠷᠠᠳ and ᠠᠷᠠᠤᠠ are one visible word, and now one shape and one canonical text.

UTN #57 and GB/T 25914-2023 keep all nine as distinct written units — their EAC vectors spell ᠠᠷᠭᠠᠯ as A A R Hx A L — and the engine still produces them. The standard's own sequence stays reachable through Shaper::shape_raw (Python: MongolianShaper._shape_raw), which is what the conformance suites compare against. It is not part of the public contract: it is #[doc(hidden)] and may change to unify further duplicates without a major bump.

shape_detailed and trace's written_by_token report each token's own units, so they are raw too — unification is a whole-word rewrite that no single token can carry. trace's shape field is the public, unified sequence.

Normalization strategy

Within the normalization table's supported written-unit domain, normalize is a pure function of shape: any two encodings that shape identically produce the same Unicode output, and the output round-trips — shape(normalize(x)) == shape(x). It is also prefix-stable. When these goals conflict the priority is round-trip > prefix-stable > shortest.

Per word:

1. shape the input into its written-unit sequence. Structural characters — MVS, nirugu, ZWJ — appear verbatim as PascalCase Mvs / Nirugu / Zwj tokens (nirugu renders a visible stem; all three are the evidence for a neighbour's init/medi/fina form). Split the shape at these tokens into chains; the tokens themselves are copied through unchanged.
2. encode each chain (right-to-left, so appending a suffix can't disturb what precedes it):
   1. partition + table lookup — the primary path. At each position take the single unit if the table has it (preferred — clean output), else the longest available multi-unit entry, and look up (position, written-unit) → (letter, FVS) in an FVS-pinned table. Each value renders its unit regardless of neighbours, so the result is a deterministic, O(N), prefix-stable function of the shape.
   2. velar-feminine refinement — a G/Gx velar's forward-coupled vowel (a/o/u) is swapped to its feminine partner (e/oe/ue) for clean output.
   3. verify — reshape the candidate in full context; accept only if it equals the target chain shape.
   4. no search fallback — the table is total over the corpus (with FVS-first selection there are no gap chains left). If an out-of-corpus shape ever misses the table, normalization fails closed with the input and the uncovered written-unit sequence. Callers may explicitly ask for the lenient variant to return the input unchanged (round-trip preserved, never a mis-encoding). A letter next to a joiner simply looks its unit up at the shifted (joined) position.
3. post-MVS suffix rule — a chain directly after MVS takes its standalone canonical (drop the MVS, normalize, re-attach), so the spelling never depends on MVS. One exception: chachlag Aa after MVS is written the bare letter a. (The isolate-I → i+FVS1 spelling is pinned in the table itself — no post-processing pass exists.)

Prefix-stability means: if word A = word B + a suffix and their shapes share a prefix, the shared region encodes identically except the single boundary unit whose position changes (final in B → medial in A). The per-unit table delivers this for free — each unit's encoding depends only on its own position, never on its neighbours.

How the table is built (the selection method): offline, a context-independence battery fills each (position, written-unit) slot with the (letter, FVS) that renders exactly that unit in every probed neighbour context (the probes include a bowed consonant, so post-bowed effects can't hide). Candidate order is letter-major, FVS-first within the letter — an FVS exists precisely to pin a form against context, so the pinned variant of the right letter always beats its context-sensitive bare form. The result is exported as JSON; the battery lives in python/scripts/gen_normalize_table.py.

Note: supported output is FVS-pinned, not bare — each unit carries the selector that fixes its form independent of context. This is what makes "same shape ⟹ same Unicode" and prefix-stability hold inside the table's domain.

The exact canonical selection policy is frozen as mng-canonical/2. It is available as Shaper::canonical_version (Python: shaper.canonical_version) and embedded in MNG.normalize.json. Applications that persist normalized search/index keys should store this version alongside them and rebuild those keys if a future release changes it.

mng-canonical/2 (0.2.0) invalidates keys stored under mng-canonical/1. Unifying the nine duplicate encodings changes canonical text: comparing current output against the base branch's 1993 mng-canonical/1 golden representatives finds 287 changed texts; three verified pairs merge into 1990 groups. Rebuild any stored normalized key. The context correction also invalidates pre-fix PR #26 output; this is not a new release or a claim that old test suites were rerun.

Repository layout

mongol-norm/
├── Cargo.toml          the crate (root package) + [workspace] members = ["python"];
│                       [workspace.package].version is the only version literal
├── Cargo.lock
├── src/                the engine — zero dependencies, MSRV 1.82, wasm32-clean
│   ├── token.rs        tokenization, structural positions
│   ├── rules.rs        the five shaping phases, one function per rule
│   ├── shaper.rs       variant resolution: shape / same_shape / shape_detailed / trace
│   ├── normalize.rs    the canonical normalizer
│   ├── written_units.rs  the written-unit and positioned-written-unit encoders
│   ├── cli.rs          the mongol-norm command (src/bin/mongol-norm.rs is a shim)
│   └── generated/      tables generated from the JSON — never hand-edited
├── tests/              the crate's integration tests and the shared fixtures
│   ├── data/           core-hud.tsv, eac-hud.tsv — vendored from mongfontbuilder
│   └── golden/         mng-canonical-v1.jsonl, mng-phase-trace-v1.json
├── README.md           this file: the GitHub landing page, the crates.io README, and a doctest
├── LICENSE  NOTICE     the crate's own licence files
├── assets/  docs/  .github/workflows/
└── python/             everything Python
    ├── Cargo.toml      the PyO3 binding crate mongol-norm-py (Rust 1.83; not on crates.io)
    ├── build.rs  src/lib.rs
    ├── pyproject.toml  maturin backend; the version comes from ../Cargo.toml
    ├── README.pypi.md  the package's PyPI long description
    ├── LICENSE  NOTICE byte-identical copies of the root files (maturin cannot reach `..`)
    ├── mongol_norm/    the package: __init__.py, _api.py (the public API),
    │   │               shaper.py (the 0.0.x compat shim), _data.py
    │   └── data/       the shaping + normalize JSON: input of the table generator,
    │                   shipped in the wheel for tooling
    ├── scripts/        gen_rust_tables.py, gen_normalize_table.py, gen_compat_goldens.py,
    │                   preprocess.py, check_dist_metadata.py
    └── tests/          the Python suite; reads the fixtures from ../../tests/{data,golden}

Data and fixtures

The shaping and normalize rules are flat, language-agnostic JSON in python/mongol_norm/data/ (MNG.json, TOD.json, SIB.json, MCH.json and MNG.normalize.json). Nothing reads it at runtime: python/scripts/gen_rust_tables.py compiles it into the static Rust tables in src/generated/, which is what both the crate and the wheel carry. The wheel still ships the JSON for tooling. The schema and the consuming algorithm are documented in docs/data-format.md, so a port in another language needs only a JSON parser.

Both test suites read the same fixtures, which live once under the crate's tests/:

Fixture	Contents
tests/data/core-hud.tsv	177 rows — mongfontbuilder's curated regression set (225 cases)
tests/data/eac-hud.tsv	3512 rows — GB/T 25914-2023 (3513 cases, 5 UTN-xfail)
tests/golden/mng-canonical-v1.jsonl	1990 canonical vectors
tests/golden/mng-phase-trace-v1.json	15 phase-trace vectors

Because the corpus and golden tests read that directory, cargo test needs a repository checkout — the published crate does not include the fixtures.

Running the tests

Rust, from the repository root:

cargo test --workspace --locked   # 256 tests: unit + corpus + goldens + properties + CLI + fuzz + this README
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo fmt --all --check
cargo package -p mongol-norm      # what crates.io would receive

Python, from python/ — the suite drives the compiled extension, so build it into a virtualenv first (needs Rust ≥ 1.83; the testing feature exposes the hook the fallback tests use). Re-run maturin develop after any Rust change:

cd python
python -m venv .venv && source .venv/bin/activate
pip install 'maturin>=1.15,<2'
maturin develop --locked --features testing        # builds mongol_norm/_native

python -m unittest discover -s tests -p 'test_*.py'   # 253 tests
python -m unittest tests.test_shaper -v               # shape / same_shape / normalize
python -m unittest tests.test_round_trip              # round-trip + canonicity + prefix-stability
python -m unittest tests.test_core_hud tests.test_eac_hud   # the upstream TSV suites
python -m unittest tests.test_rust_twin               # tables fresh, versions in lockstep

The generators are run from the repository root; each --check fails when the committed output differs from what it would generate now (all three run in CI):

python python/scripts/gen_rust_tables.py --check
python python/scripts/gen_normalize_table.py --check
python python/scripts/gen_compat_goldens.py --check

Current totals: 256 Rust tests (unit + property + 177 core-hud and 3512 eac-hud corpus rows, 1993 canonical and 15 phase-trace golden vectors, fuzz, and this README's doctests) and 253 Python tests, green on Rust stable / 1.82 (the core crate's MSRV; the binding crate needs 1.83) and CPython 3.9 – 3.14.

Use cases

• Search & retrieval — index Mongolian text with unique keys per visual word
• Deduplication — detect identical words encoded differently
• Spell checking — normalize before dictionary lookup
• Corpus linguistics — consistent word frequency counts
• OCR post-processing — standardize OCR output that may use inconsistent encodings
• Input method engines — validate and normalize user input

Supported locales

Locale	Script	Status
MNG	Hudum (Traditional Mongolian)	✅ Full shaping + normalization
TOD	Todo	⬜ Shaping rules generated, normalization WIP
SIB	Sibe	⬜ Shaping rules generated, normalization WIP
MCH	Manchu	⬜ Shaping rules generated, normalization WIP

Documentation

• docs.rs/mongol-norm — the full crate API
• docs/data-format.md — the JSON schema and the normalize algorithm, for ports in other languages
• python/README.pypi.md — the Python package
• docs/releasing.md — how a release is cut
• docs/superpowers/specs/2026-09-01-rust-core-design.md — the engine design and fidelity contract
• docs/superpowers/specs/2026-09-02-python-bindings-design.md — the Python bindings design

Data sources & acknowledgments

• UTN #57 v4 — Unicode Technical Note: Encoding and Shaping of the Mongolian Script. The authoritative specification for Mongolian shaping rules.
• mongfontbuilder by Kushim Jiang — source of the bundled flat variant tables in python/mongol_norm/data/ (preprocessed from data.variants / data.particles) and of the core-hud.tsv / eac-hud.tsv regression suites vendored into tests/data/. Both UTN #57 and mongfontbuilder are authored by the same person.
• GB/T 25914—2023 — China national standard for Traditional Mongolian nominal characters; source of the EAC compliance test set.
• Claude Code — this project was developed with AI assistance. The shaping rules are derived from the above sources; Claude Code was used to implement and structure the engine.

License

MIT License — see LICENSE.

The shaping rules and bundled data are derived from mongfontbuilder (MIT) and UTN #57. Their required notices are retained in NOTICE.

---

中文

mongol-norm 是传统蒙古文（回鹘式，Hudum）的形态感知规范化器：完整实现 UTN #57 v4 整形流程，以及 FVS 钉死的 canonical 规范化器；零依赖 Rust crate，同时可编译到 wasm32-unknown-unknown。

同一个可见的蒙古文词可以写成多种不同的 Unicode 序列，这会破坏搜索、去重和索引。shape() 把词渲染成 书写单元序列——即字体会画出的形状，但无需字体——normalize() 则把同一个词的任意编码改写成该形态唯一的 canonical 编码。

crate 就是引擎，位于仓库根目录。python/ 下是它的一层薄 PyO3 绑定，以 mongol-norm 发布到 PyPI；二者共用一个版本字面量、一套数据表和 一套固件，结果逐字节相同——见 Python 包。

状态

库的两部分——整形与规范化（MNG / Hudum）——每次 push 都对照同一批上游语料验证，覆盖 Rust stable 与 MSRV、 以及 CPython 3.9 – 3.14。

✅ Shaping（整形）

shape() 和 same_shape() 对照两套上游 TSV 套件交叉验证：

套件	用例数	通过	说明
mongfontbuilder/core-hud.tsv	225	100%	精选回归集
mongfontbuilder/eac-hud.tsv (GB/T 25914-2023)	3513	100%	5 个 UTN ↔ EAC 分歧 case 跳过（跟 mongfontbuilder 自己的 pytest.mark.xfail 列表一致）
手写单元测试	—	100%	shape / same_shape / joiner token（nirugu、ZWJ）

两套 TSV 用的都是国标自己的书写单元序列，因此对照 shape_raw 检查——即引擎统一九个重复编码之前的输出 （见重复编码）。3512 行 EAC 中有 375 行、177 行 core 中有 30 行，公开 shape 与国标的拼法不同。

✅ Normalization（规范化）— 语料域保证，机器验证

对于内置规范化表覆盖的 written-unit shape，normalize / normalize_text / normalize_written_units 是 shape 的纯函数，以下不变量在 CI 中对每一条语料编码逐一验证：

性质	结果
往返 —— shape(normalize(x)) == shape(x)	3757 / 3757 语料编码（100%）
同形同码 —— shape 相同 ⟹ 输出 Unicode 相同	1990 / 1990 shape 组（100%）
前缀稳定 —— 词与词+后缀共享前缀编码	2240 / 2240 真实语料词对（100%）

范围说明：规范化目前只实现了 MNG（Hudum）—— Todo / 锡伯文 / 满文已加载 shaping 规则，尚无规范化。上述保证 覆盖已检查语料及内置表可编码的 written-unit chain。语料外 chain 若未被覆盖，默认 fail closed，返回 Error::NormalizationFallback；只有明确接受原样回退时才调用 normalize_allow_fallback。

本项目由 Claude Code（AI 辅助编码）生成；测试与部分核心代码经人工审核，测试 覆盖比较充分（语料往返 / 同形同码 / 前缀稳定 + 上游跨实现套件）。当前为预览版，正常使用应无问题；遇到 问题欢迎提 issue 和 PR。Shaping 逻辑源自 UTN #57 v4 和 mongfontbuilder。

安装

cargo add mongol-norm

或手写：

[dependencies]
mongol-norm = "0.2.0"

命令行工具装成独立二进制：

cargo install mongol-norm

MSRV 为 1.82。API 文档见 docs.rs/mongol-norm。

使用方法

use mongol_norm::{Error, Locale, PositionedWrittenUnit, Shaper, UnitPosition, WrittenUnit};

fn main() -> Result<(), Error> {
    let shaper = Shaper::new(Locale::Mng);

    // 字形化：书写单元序列
    assert_eq!(shaper.shape("ᠰᠠᠢᠨ")?.len(), 5); // [S, A, I, I, A]
    assert_eq!(shaper.shape_str("ᠰᠠᠢᠨ")?, "S+A+I+I+A");

    // 比较：两个编码视觉上是否相同？
    assert!(shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")?);

    // 规范化为唯一的 FVS 钉死 canonical 编码（严格模式；未覆盖的 shape 会报错，
    // `normalize_allow_fallback` 则原样返回）
    let canonical = shaper.normalize("ᠰᠡᠢᠨ")?;
    assert_eq!(shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")?, canonical);

    // 自由文本：只规范化蒙古文词，其余原样保留
    let text = shaper.normalize_text("Hello ᠰᠡᠢᠨ world")?;
    assert!(text.starts_with("Hello ") && text.ends_with(" world"));

    // 书写单元与权威 HUD position
    assert_eq!(
        shaper.normalize_written_units(&[WrittenUnit::B, WrittenUnit::Aa])?,
        "ᠪᠠ᠋"
    );
    let records = [
        PositionedWrittenUnit::new(WrittenUnit::B, UnitPosition::Init),
        PositionedWrittenUnit::new(WrittenUnit::Aa, UnitPosition::Fina),
    ];
    assert_eq!(shaper.normalize_positioned_written_units(&records)?, "ᠪᠠ᠋");
    assert_eq!(shaper.canonical_version(), Some("mng-canonical/2"));

    // 公开 shape 统一了九个重复编码，所以 ᠠᠷᠠᠳ 与 ᠠᠷᠠᠤᠠ ——同一个可见词的两种拼法——
    // 是同一个 shape、同一个 canonical 文本
    assert_eq!(shaper.shape_str("ᠠᠷᠠᠳ")?, "A+A+R+A+O+A");
    assert!(shaper.same_shape("ᠠᠷᠠᠳ", "ᠠᠷᠠᠤᠠ")?);
    assert_eq!(shaper.normalize("ᠠᠷᠠᠳ")?, shaper.normalize("ᠠᠷᠠᠤᠠ")?);
    Ok(())
}

shape_detailed 返回逐 token 的细节（码位、alias、position、FVS、condition、书写单元），trace 返回 phase-trace golden 固件背后的逐规则 condition 变化。完整 API 见 docs.rs。

命令行

crate 自带 mongol-norm 二进制——Python 包安装的 mongol-norm 命令就是它。shape 输出词渲染出的书写 单元序列，normalize 把同一个词的任意编码统一成唯一的 canonical（FVS 钉死）形式，normalize-text 只 规范化自由文本中的蒙古文词，normalize-written-units 编码已 shape 的书写单元，same 判断两种编码是否 同形：

cargo install mongol-norm

# 直接传文本
mongol-norm shape 'ᠰᠠᠢᠨ'                        # → S+A+I+I+A（渲染出的书写单元序列）
mongol-norm normalize 'ᠰᠡᠢᠨ'                    # 单词 → canonical 编码
mongol-norm normalize --allow-fallback 'ᠰᠡᠢᠨ'   # 未覆盖时原样返回
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'         # 混合文字：只规范化蒙古文词
mongol-norm normalize-written-units 'B+Aa'      # → ᠪᠠ᠋
mongol-norm normalize-written-units 'BZwj'      # 紧凑 PascalCase 单元串
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'                  # 同形退出码 0，不同为 1

# 管道 / 标准输入（文本位置写 `-`）
echo 'B+Aa' | mongol-norm normalize-written-units -
cat doc.txt | mongol-norm normalize-text -

# 文件输入 / 输出
mongol-norm normalize-text -i in.txt -o out.txt

# 批量：一行一词输入，一行一个 canonical 输出
mongol-norm normalize --batch -i words.txt -o canonical.txt
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize --batch -
mongol-norm normalize-written-units --batch -i units.txt -o canonical.txt

--locale MNG|TOD|SIB|MCH 选文种（默认 MNG，只有 MNG 支持规范化）。它是全局选项，要写在子命令 之前——mongol-norm --locale TOD shape 'ᠰᠠᠢᠨ'，不能写在子命令后面。--allow-fallback 原样保留未覆盖 的词，-- 结束选项（其后的 - 当文本），出错退出码 2（same 打印 true/false，退出码 0/1）， mongol-norm --help 列出全部参数。

shape 和 normalize 处理单个词，拒绝蒙古文词字母表之外的任何字符——包括 echo 追加的换行，所以管道 请用 printf，或者用 --batch（一行一词），自由文本请用 normalize-text。

normalize-written-units 接受紧凑 PascalCase 或显式 + 边界。紧凑输入必须只有一种合法切分；存在歧义时 fail closed，须改用 +。解析后继续执行与 normalize_written_units API 相同的 exact-shape 校验，因此语法 合法的 unit stream 若没有 canonical MNG 编码仍会被拒绝。

Python 包

pip install mongol-norm

PyPI 包是本 crate 之上的一层薄 PyO3 封装，由 maturin 从 python/ 构建。引擎及其数据表已编译进 mongol_norm._native 扩展，因此零运行时依赖，也不需要安装 Rust 工具链。

预编译的 cp39-abi3 wheel（每个平台一个，服务全部 CPython ≥ 3.9）覆盖 Linux x86_64 与 aarch64 （glibc 走 manylinux2014，musl 走 musllinux_1_2）、macOS x86_64 与 Apple silicon、Windows x64；CI 实测 CPython 3.9 – 3.14。其他平台 pip 会回退到源码包（sdist），在本地编译扩展，需要 Rust ≥ 1.83 工具链 （maturin 构建后端由 pip 自动获取）。

from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")

shaper.shape("ᠰᠠᠢᠨ")                        # → ['S', 'A', 'I', 'I', 'A']
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")           # → True
shaper.normalize("ᠰᠡᠢᠨ")                    # → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")   # → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'
shaper.normalize_written_units(["B", "Aa"]) # → 'ᠪᠠ᠋'

mongol-norm 控制台脚本就是 crate 的 CLI，在进程内运行。完整的 Python 文档——全部 API、异常类型、书写 单元编码器与命令行——见 python/README.pypi.md，它同时 是该包的 PyPI 页面。

从 0.0.x 升级。 公开 API（MongolianShaper、NormalizationFallbackError、mongol-norm 命令）不变； from mongol_norm.shaper import … 与 python -m mongol_norm.shaper … 通过兼容 shim 仍可用。差异：需要 Python ≥ 3.9；未知 locale 抛 ValueError（原为 FileNotFoundError）；mongol_norm.rules 及 shaper 私有 内部实现已移除；命令行改为 crate 自带的 CLI（新增 -V/--version；其余刻意差异见 src/cli.rs 顶部）。 新增：trace()、rule_names()、parse_written_units()。

为什么做这个项目

传统蒙古文在 Unicode 中存在一个根本性问题：同一个可见词形可以用多种不同的 Unicode 序列编码。原因是：

1. 字母共享字形 — A 和 E 在中间和尾部位置外形完全相同；O/U、OE/UE 共享形态；QA 和 GA 根据元音和谐共享形态。
2. 多种编码路径 — 同一个齿形字形可以编码为 I、YA+FVS1，甚至两个独立的 I 字符。
3. 冗余的 FVS 使用 — 自由变体选择符（FVS1–FVS4）可以创建渲染结果完全相同的等价序列。
4. 连接控制符 — nirugu（U+180A，可见的连笔延长符）和 ZWJ（U+200D）会强制字母取连接形，而连接语境下更多字母塌缩成同一字形（nirugu+o 和 nirugu+u 渲染完全相同）。
5. MVS/NNBSP 后的后缀词 — 同一个渲染出的后缀可以用不同字母拼写（MVS+a 和 MVS+e 都渲染 chachlag 形；MVS+u 和 MVS+ue 同形）。

这意味着：

• 搜索失效：搜索同一个词的某种编码，找不到另一种编码，尽管它们外形完全一样。
• 去重失败：同一个词有多种 Unicode 表示。
• 索引不可靠：同一个词的不同编码产生不同的索引键。

mongol-norm 使用完整的 UTN #57 v4 shaping 过程（5 步条件映射）对输入进行字形化，通过比较字形序列 检测视觉上相同的词形，并把受支持的 shape 规范化为唯一的、FVS 钉死的 canonical Unicode 编码—— 同 shape ⟹ 同 Unicode，并精确往返还原。

示例：以下五种编码都表示 "sain"（好的），外形完全相同：

工作原理

本规范化器实现了一个轻量级蒙古文 shaping 引擎——功能相当于 HarfBuzz 配合字体文件所做的事情，但仅使用 UTN #57 v4 的规则数据和 mongfontbuilder 项目的变体数据。不需要字体文件。

Shaping 管线（UTN #57 v4 蒙古文特定阶段）

步骤	名称	说明
1	Chachlag	MVS（蒙古文元音分隔符）后的 a/e 后缀形态
2	Syllabic	辅音/元音上下文：onset/devsger/marked/阴阳和谐/dotless
3	Particle	MVS 小品词词典查找
4	Devsger	元音后的 i 获得双齿形态：I → I+I（vowel_devsger）
5	Post-bowed	弓形辅音（G/B/K/P/F）后的元音形态变化

重复编码

shape 承诺是可见词的指纹——“同 shape ⟹ 同 normalize”正是这个 crate 存在的理由。有九个书写单元 打破了这个承诺：它们与另一串单元渲染出完全相同的墨迹，导致同一个词的两种拼法 shape 不同。

其中五个通过展开（单个单元 → 单元对）统一：

重复编码	公开 shape	见证词对
Dd:medi	O:medi A:medi	ᠣᠳᠪᠣ / ᠠ᠋ᠣᠣᠠᠪᠣ᠋
Dd:fina	O:medi A:fina	ᠠᠷᠠᠳ / ᠠᠷᠠᠤᠠ
H:medi	A:medi A:medi	ᠪᠠᠭᠰᠢ / ᠪᠠᠠᠠᠰᠢ
Hx:medi	N:medi N:medi	ᠠᠷᠭᠠᠯ / ᠠ᠋ᠠᠷᠨ᠋ᠨ᠋ᠠᠯ
Cr:init	O:init O:medi	ᡂ᠊ / ᠤ᠋ᠤ᠊

另外四种形态采用收缩，仅在验证过的位置和上下文中选择较短的 canonical 形式。 Aa:fina 紧邻圆头书写单位时有齿，非圆头书写单位之后无齿；反复插入或删除 A 并不保持墨迹：

单元对	公开 shape	见证词对
独占整条 chain 的 A Aa	A:isol	ᠡ / ᠡᠠ᠋
bowed written unit A:medi Aa:fina	bowed written unit Aa:fina	ᠪᠠ / ᠪᠠᠠ᠋
位于 chain 末尾的 O Aa	B2:fina	᠊ᠪ᠋ / ᠊ᠤᠠ᠋
位于 chain 末尾的 I Aa	G:fina	᠊ᠭ / ᠊ᠢᠠ᠋

每条规则都带位置条件，用的是 normalize 本来就在用的 chain 位置——以结构单元切分的槽位，nirugu 或 ZWJ 邻居像它影响渲染那样为 chain 补位。这就是 B2 与 G 的见证词要带前导 nirugu 的原因：正是它让该单元成为词末。 未被见证的形态一律不动——词首/词末的 H/Hx 是不同的墨迹，单独一个 Cr 的 chain 是 Cr:isol 而非被见证的 Cr:init，单独一个 A 的 chain 本就是 canonical。

术语。 UTN #57 使用 bowed written units（圆头书写单位），并将相应 lookup 称为 Post-bowed；参见 UTN #57 第 4 修订版。

上下文。 Hudum 的完整圆头书写单位集合是 B、P、F、G、Gx、K、K2。来源为 Hudum 书写单元连写变体表、 上游 required ligatures 数据 中的 BAa/PAa/FAa/GAa/GxAa/KAa/K2Aa，以及本项目 src/rules.rs 的 post-bowed 类。 圆头书写单位必须在同一 chain 内紧邻 A:medi：B A Aa → B Aa，但 N A Aa、A A Aa、 B A A Aa 均保留。Joiner 补位不是圆头书写单位，规则不能跨结构单元。独立整链规则 A:init Aa:fina → A:isol 不变。

终止与幂等性。 一次展开后只检查一次尾部单元对。展开不产生新的展开目标；收缩也不会让 词首/词末 H/Hx 变成词中。所有收缩都位于链尾：结果 A/B2/G 不以 Aa 结尾，而收缩得到的 Aa 前面是圆头书写单位、不是 A/O/I，不能再收缩。幂等性来自正确上下文，不是强制跑到不动点。 20 单元字母表（含全部圆头书写单位、展开目标和结构单元）上长度 ≤ 4 的 168 421 个输入均已穷举验证， 全部语料的书写单元再编码也可还原形状。

展开不能授权非圆头书写单位收缩：ᠲᠡᠳ᠌ᠡ᠋ (T A Dd Aa) 应为 T A O A Aa，不是 T A B2， 因为 O 不是圆头书写单位；B H Aa 展开后同样保留 B A A Aa。两种交互均有回归测试。

面向用户的一切都看到统一后的序列：same_shape、normalize 和书写单元编码器。 normalize_written_units 仍然接受重复编码作为输入并统一它们，调用方已有的数据继续可用。 ᠠᠷᠠᠳ 与 ᠠᠷᠠᠤᠠ 是同一个可见词，现在也是同一个 shape、同一个 canonical 文本。

UTN #57 与 GB/T 25914-2023 把这九个保留为不同的书写单元——其 EAC 向量把 ᠠᠷᠭᠠᠯ 拼作 A A R Hx A L—— 引擎也仍然产出它们。国标自己的序列通过 Shaper::shape_raw（Python：MongolianShaper._shape_raw）依然可达， 一致性套件比对的就是它。它不属于公开契约：标了 #[doc(hidden)]，将来可能在不升 major 的情况下统一更多 重复编码。

shape_detailed 与 trace 的 written_by_token 报告的是每个 token 自身的单元，因此也是原始序列——统一是 整词级的改写，单个 token 承载不了。trace 的 shape 字段则是公开的统一序列。

规范化策略

在规范化表支持的 written-unit 域内，normalize 是 shape 的纯函数：任意两个 shape 相同的编码， normalize 输出相同，且往返成立 —— shape(normalize(x)) == shape(x)，同时前缀稳定。三者冲突时优先级： 往返 > 前缀稳定 > 最短。

逐词：

1. shape 成书写单元序列。结构字符 —— MVS、nirugu、ZWJ —— 原样输出为 PascalCase Mvs / Nirugu / Zwj token（nirugu 是可见的连笔字形；三者都是邻居字母 init/medi/fina 形的依据）。按这些 token 切成 chain，token 本身原样拷贝。
2. 逐 chain 编码（从右往左，这样加后缀不影响前面）：
   1. 划分 + 查表（主路径）：每个位置优先取单单元（输出干净），否则取最长多单元，查 (位置, 书写单元) → (字母, FVS) 的 FVS 钉死表。每个值不依赖邻居就渲染出该单元 → 确定性、O(N)、前缀稳定。
   2. velar 阴性微调：G/Gx 前向耦合的元音（a/o/u）换成阴性（e/oe/ue），输出更干净。
   3. 校验：在完整上下文里重新 shape，只接受与目标 chain shape 一致的结果。
   4. 没有搜索兜底：FVS 优先的选择下，表对全部语料 chain 是完备的（缺口为零）。语料外的 shape 万一查不到表，默认 fail closed，带上原输入与未覆盖的 written-unit 序列；只有明确接受原样回退时才调用 lenient 变体（保住往返，绝不错编）。紧邻 joiner 的字母只是按移动后的连接位置查表。
3. MVS 后缀规则：紧跟 MVS 的 chain 用其 standalone canonical（去掉 MVS、归一、再拼回），拼写不依赖 MVS。唯一例外：MVS 后的 chachlag Aa 写裸字母 a。（孤立 I → i+FVS1 的拼写已钉进表本身 —— 不存在后处理。）

前缀稳定的含义：若词 A = 词 B + 后缀，且二者 shape 共享前缀，则共享部分编码完全一致，只有那个位置 发生变化的边界单元不同（在 B 里是词尾、在 A 里变词中）。逐单元表天然保证这点——每个单元的编码只取决 于它自己的位置，与邻居无关。

表是怎么来的（选择方法）：离线跑一个 context 无关性电池——对每个 (位置, 书写单元)，挑出在 所有探测邻居上下文里都恰好渲染出该单元的 (字母, FVS)（探针含弓形辅音，post-bowed 效应藏不住）。 候选顺序是字母优先、字母内 FVS 优先——FVS 的意义就是把字形从 context 里隔离出来，所以正确字母的钉死形 永远优于其受感染的裸形。结果导出成 JSON；电池在 python/scripts/gen_normalize_table.py。

注意：受支持输出是 FVS 钉死而非 bare —— 每个单元都带着把字形固定住、不受上下文影响的选择符，这正是 “同 shape ⟹ 同 Unicode”和前缀稳定在表内成立的原因。

当前精确 canonical 选择策略冻结为 mng-canonical/2。可通过 Shaper::canonical_version（Python： shaper.canonical_version）读取，并写入 MNG.normalize.json。持久化规范化搜索键/索引键的应用应同时保存 该版本；未来版本若发生变化，应重建这些键。

mng-canonical/2（0.2.0）会使 mng-canonical/1 下存储的键失效。 当前输出与 base 分支 1993 个 mng-canonical/1 golden 代表词比较，287 个 canonical 文本改变；三对验证过的重复组 合并成 1990 组。已存储的规范化键必须重建，上下文修复前 PR #26 的输出也受影响。 这是 fixture 对比实测，不表示重新运行过旧测试，也不是新版本发布。

仓库结构

mongol-norm/
├── Cargo.toml          crate（根 package）+ [workspace] members = ["python"]；
│                       [workspace.package].version 是唯一的版本字面量
├── Cargo.lock
├── src/                引擎——零依赖，MSRV 1.82，可编译到 wasm32
│   ├── token.rs        分词、结构位置
│   ├── rules.rs        五个 shaping 阶段，一条规则一个函数
│   ├── shaper.rs       变体解析：shape / same_shape / shape_detailed / trace
│   ├── normalize.rs    canonical 规范化算法
│   ├── written_units.rs  书写单元 / 带位置书写单元编码器
│   ├── cli.rs          mongol-norm 命令（src/bin/mongol-norm.rs 只是壳）
│   └── generated/      由 JSON 生成的数据表——请勿手改
├── tests/              crate 的集成测试与共享固件
│   ├── data/           core-hud.tsv、eac-hud.tsv —— 来自 mongfontbuilder
│   └── golden/         mng-canonical-v1.jsonl、mng-phase-trace-v1.json
├── README.md           本文件：GitHub 首页、crates.io README，同时是 doctest
├── LICENSE  NOTICE     crate 自己的许可文件
├── assets/  docs/  .github/workflows/
└── python/             全部 Python 内容
    ├── Cargo.toml      PyO3 绑定 crate mongol-norm-py（需 Rust 1.83；不发布到 crates.io）
    ├── build.rs  src/lib.rs
    ├── pyproject.toml  maturin 后端；版本从 ../Cargo.toml 读取
    ├── README.pypi.md  包在 PyPI 上的长描述
    ├── LICENSE  NOTICE 根目录同名文件的逐字节副本（maturin 不能引用 `..`）
    ├── mongol_norm/    包本体：__init__.py、_api.py（公开 API）、
    │   │               shaper.py（0.0.x 兼容 shim）、_data.py
    │   └── data/       shaping + normalize JSON：表生成器的输入，
    │                   随 wheel 发布供工具脚本使用
    ├── scripts/        gen_rust_tables.py、gen_normalize_table.py、gen_compat_goldens.py、
    │                   preprocess.py、check_dist_metadata.py
    └── tests/          Python 套件；固件从 ../../tests/{data,golden} 读取

数据与固件

shaping 与 normalize 规则是扁平、语言无关的 JSON，位于 python/mongol_norm/data/（MNG.json、 TOD.json、SIB.json、MCH.json 与 MNG.normalize.json）。运行时不读它： python/scripts/gen_rust_tables.py 把它编译成 src/generated/ 下的静态 Rust 表，crate 与 wheel 携带的都是 这些表；wheel 仍随包发布 JSON，供工具脚本使用。schema 与消费算法见 docs/data-format.md，其他语言只需 一个 JSON 解析器即可移植。

两套测试读同一批固件，它们只存一份，在 crate 的 tests/ 下：

固件	内容
tests/data/core-hud.tsv	177 行 —— mongfontbuilder 的精选回归集（225 个 case）
tests/data/eac-hud.tsv	3512 行 —— GB/T 25914-2023（3513 个 case，5 个 UTN-xfail）
tests/golden/mng-canonical-v1.jsonl	1990 个 canonical 向量
tests/golden/mng-phase-trace-v1.json	15 个 phase-trace 向量

因为语料与 golden 测试读取这个目录，cargo test 需要仓库 checkout —— 发布的 crate 不包含固件。

运行测试

Rust，在仓库根目录：

cargo test --workspace --locked   # 256 个测试：单元 + 语料 + golden + 性质 + CLI + fuzz + 本 README
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo fmt --all --check
cargo package -p mongol-norm      # crates.io 实际收到的内容

Python，在 python/ 下——套件跑的是编译好的扩展，所以先在 virtualenv 里把它构建出来（需要 Rust ≥ 1.83； testing feature 暴露 fallback 测试用的钩子）。任何 Rust 改动之后都要重跑 maturin develop：

cd python
python -m venv .venv && source .venv/bin/activate
pip install 'maturin>=1.15,<2'
maturin develop --locked --features testing        # 构建 mongol_norm/_native

python -m unittest discover -s tests -p 'test_*.py'   # 253 个测试
python -m unittest tests.test_shaper -v               # shape / same_shape / normalize
python -m unittest tests.test_round_trip              # 往返 + 同形同码 + 前缀稳定
python -m unittest tests.test_core_hud tests.test_eac_hud   # 上游 TSV 套件
python -m unittest tests.test_rust_twin               # 生成表新鲜、版本一致

生成脚本在仓库根目录运行；每个 --check 在 committed 文件与当前应生成结果不一致时失败（三个都在 CI 里跑）：

python python/scripts/gen_rust_tables.py --check
python python/scripts/gen_normalize_table.py --check
python python/scripts/gen_compat_goldens.py --check

当前总数：256 个 Rust 测试（单元 + 性质 + 177 条 core-hud 与 3512 条 eac-hud 语料行、1993 个 canonical 与 15 个 phase-trace golden 向量、fuzz，以及本 README 的 doctest）和 253 个 Python 测试，在 Rust stable / 1.82（核心 crate 的 MSRV；绑定 crate 需要 1.83）与 CPython 3.9 – 3.14 上全绿。

应用场景

• 搜索与检索 — 为每个可见词形建立唯一索引键
• 文本去重 — 检测编码不同但外形相同的词
• 拼写检查 — 规范化后再查词典
• 语料库语言学 — 一致的词频统计
• OCR 后处理 — 标准化可能使用不一致编码的 OCR 输出
• 输入法引擎 — 验证和规范化用户输入

支持的语种

Locale	文字	状态
MNG	Hudum（传统蒙文）	✅ 完整 shaping + 规范化
TOD	Todo（托忒文）	⬜ shaping 规则已生成，规范化开发中
SIB	Sibe（锡伯文）	⬜ shaping 规则已生成，规范化开发中
MCH	Manchu（满文）	⬜ shaping 规则已生成，规范化开发中

文档

• docs.rs/mongol-norm —— 完整的 crate API
• docs/data-format.md —— JSON schema 与 normalize 算法，供其他语言移植
• python/README.pypi.md —— Python 包
• docs/releasing.md —— 发版流程
• docs/superpowers/specs/2026-09-01-rust-core-design.md —— 引擎设计与保真约定
• docs/superpowers/specs/2026-09-02-python-bindings-design.md —— Python 绑定设计

数据来源与致谢

• UTN #57 v4 — Unicode 技术注释：蒙古文编码与字形化。蒙古文 shaping 规则的权威规范。
• mongfontbuilder（Kushim Jiang）— python/mongol_norm/data/ 内置扁平变体表的来源（从 data.variants / data.particles 预处理而来），同时也是 vendor 进 tests/data/ 的 core-hud.tsv / eac-hud.tsv 回归套件的来源。UTN #57 和 mongfontbuilder 的作者是同一人。
• GB/T 25914—2023 — 中国国家标准：传统蒙古文名义字符、表现字符和控制字符使用规则；EAC 一致性测试集的来源。
• Claude Code — 本项目使用 AI 辅助开发。shaping 规则来源于上述数据；Claude Code 用于实现和组织引擎代码。

许可证

MIT License —— 见 LICENSE。

整形规则与内置数据派生自 mongfontbuilder（MIT）和 UTN #57，其许可证要求的署名保留在 NOTICE 中。