mongol-norm 0.3.0

# mongol-norm

[![crates.io](https://img.shields.io/crates/v/mongol-norm.svg)](https://crates.io/crates/mongol-norm)
[![docs.rs](https://img.shields.io/docsrs/mongol-norm)](https://docs.rs/mongol-norm)
[![PyPI](https://img.shields.io/pypi/v/mongol-norm.svg)](https://pypi.org/project/mongol-norm/)
[![Test](https://github.com/Satsrag/mongol-norm/actions/workflows/test.yml/badge.svg)](https://github.com/Satsrag/mongol-norm/actions/workflows/test.yml)

[English](#english) | [中文](#中文)

---

<a id="english"></a>
## English

> [!WARNING]
> **Beta.** `shape` / `same_shape` should be stable. The `normalize` output currently encodes by
> **glyph shape**, not the standard **phonetic (nominal-character) spelling**, and may change in a
> later release. For example ᠮᠣᠩᠭᠣᠯ (`MA+O+ANG+GA+O+LA`) normalizes to ᠮᠣᠠᠭ᠌ᠨ᠋ᠨ᠋ᠣᠯ
> (`MA+O+A+GA+FVS2+NA+FVS1+NA+FVS1+O+LA`). If you store normalized keys, store
> `canonical_version()` with them and rebuild when it changes.

**mongol-norm** tells whether two Traditional Mongolian (Hudum) strings look the same, and maps
them to one key. Rust crate with zero dependencies, plus a Python package and a CLI.

### Why

The same visible Mongolian word can be typed as many different Unicode sequences — A/E look alike
in medial and final position, O/U and OE/UE share forms, FVS selectors and joiners (nirugu, ZWJ)
can produce the same glyph in several ways. So search misses matches, deduplication fails and
index keys differ for words that look identical.

![Five encodings of "sain" all normalizing to the same canonical form](https://raw.githubusercontent.com/Satsrag/mongol-norm/main/assets/sain-variants.png)

mongol-norm runs the [UTN #57 v4](https://www.unicode.org/notes/tn57/tn57-4.html) shaping rules
(no font needed) to get the *written units* a font would draw:

- `shape(word)` — the written-unit sequence, a fingerprint of the visible word
- `same_shape(a, b)` — do two encodings render identically?
- `normalize(word)` / `normalize_text(text)` — one canonical Unicode string per shape

Use it for search and indexing, deduplication, corpus counts, spell-check lookup, and cleaning OCR
or input-method output.

### Web playground

**[Try it online →](https://www.satsrag.dev/norm/)** — enter Mongolian text to see `shape`,
`norm`, Unicode code points, and vertical previews. Everything runs in your browser.

The [web playground](https://github.com/Satsrag/mongol-norm/blob/main/web/README.md) runs the Rust engine and renders input and
normalized text with the bundled `hudum.otf`. From this repository, run
`npm --prefix web run dev`, then open <http://127.0.0.1:4173> (see the web README for build prerequisites).

### Install

**Rust** (MSRV 1.82, zero dependencies, also builds for `wasm32-unknown-unknown`):

```bash
cargo add mongol-norm
```

or in `Cargo.toml`:

```toml
[dependencies]
mongol-norm = "0.3.0"
```

**Python** (CPython ≥ 3.9, no runtime dependencies, no Rust toolchain needed):

```bash
pip install mongol-norm
```

Prebuilt wheels cover Linux x86_64 / aarch64 (glibc and musl), macOS x86_64 / Apple silicon and
Windows x64. On other platforms pip builds from the source distribution, which needs Rust ≥ 1.83.

**Command line** — either of these puts `mongol-norm` on `PATH`:

```bash
cargo install mongol-norm
pip install mongol-norm
```

### Rust

```rust
use mongol_norm::{Error, Locale, Shaper};

fn main() -> Result<(), Error> {
    let shaper = Shaper::new(Locale::Mng);

    // Shape: the written units a font would draw
    assert_eq!(shaper.shape_str("ᠰᠠᠢᠨ")?, "S+A+I+I+A");

    // Compare: two encodings, one visible word
    assert!(shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")?);

    // Normalize: every encoding of a shape → the same string
    assert_eq!(shaper.normalize("ᠰᠡᠢᠨ")?, shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")?);

    // Free-form text: Mongolian words normalized, everything else kept
    let text = shaper.normalize_text("Hello ᠰᠡᠢᠨ world")?;
    assert!(text.starts_with("Hello ") && text.ends_with(" world"));
    Ok(())
}
```

Full API on [docs.rs](https://docs.rs/mongol-norm).

**Font backend** (`features = ["font"]`, Rust 1.85): the same `shape`, but computed by
[harfrust](https://github.com/harfbuzz/harfrust) (the Rust port of HarfBuzz) running the
Hudum font mongfontbuilder composes, bundled without its outlines (30 KB), instead of the
ported rules — what the reference font draws:

```rust
# #[cfg(feature = "font")] {
use mongol_norm::FontShaper;

let font = FontShaper::new();
assert_eq!(font.shape_str("ᠰᠠᠢᠨ")?, "S+A+I+I+A");
# }
# Ok::<(), mongol_norm::Error>(())
```

The default build stays dependency-free; `normalize` always uses the rule engine.

### Python

```python
from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")

shaper.shape("ᠰᠠᠢᠨ")                        # → ['S', 'A', 'I', 'I', 'A']
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")           # → True
shaper.normalize("ᠰᠡᠢᠨ")                    # → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")   # → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'

# Deduplicate
{shaper.normalize(w) for w in ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]}   # → {'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}
```

More in the [Python README](https://github.com/Satsrag/mongol-norm/blob/main/python/README.pypi.md).

### Command line

```bash
mongol-norm shape 'ᠰᠠᠢᠨ'                  # → S+A+I+I+A
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'            # exit 0 if identical, 1 if not
mongol-norm normalize 'ᠰᠡᠢᠨ'              # one word
mongol-norm normalize-text -i in.txt -o out.txt
mongol-norm normalize --batch -i words.txt -o keys.txt   # one word per line
```

`mongol-norm --help` lists every option.

### Notes

- **Hudum** (`MNG`) only. Todo, Sibe and Manchu are not supported.
- A word whose shape the built-in table cannot encode fails with an error rather than being
  guessed; `normalize_allow_fallback` (Python: `strict=False`) returns it unchanged instead.
- Shaping passes 100% of mongfontbuilder's `core-hud` and GB/T 25914-2023 `eac-hud` suites;
  normalization round-trips every corpus word (`shape(normalize(x)) == shape(x)`). The font
  backend passes the Hudum suites of the mongfontbuilder commit it is pinned to, with the EAC
  rows mongfontbuilder itself marks as expected failures, and shapes every corpus word like the
  rule engine but one (`XIM11-46`, one of those rows).
- This project was written with [Claude Code](https://claude.ai/code); tests and core code were
  reviewed by hand. Please report problems as [issues](https://github.com/Satsrag/mongol-norm/issues).

### More

- [`docs/internals.md`](https://github.com/Satsrag/mongol-norm/blob/main/docs/internals.md) — shaping pipeline, duplicate encodings, normalization algorithm
- [`docs/data-format.md`](https://github.com/Satsrag/mongol-norm/blob/main/docs/data-format.md) — JSON rule tables, for ports to other languages
- [`docs/development.md`](https://github.com/Satsrag/mongol-norm/blob/main/docs/development.md) — repository layout, running the tests
- [`docs/releasing.md`](https://github.com/Satsrag/mongol-norm/blob/main/docs/releasing.md) — cutting a release

### License

MIT — see [LICENSE](https://github.com/Satsrag/mongol-norm/blob/main/LICENSE). Shaping rules and
data are derived from [UTN #57](https://www.unicode.org/notes/tn57/tn57-4.html) and
[mongfontbuilder](https://github.com/Kushim-Jiang/mongfontbuilder) (MIT) by Kushim Jiang; test
suites from mongfontbuilder and GB/T 25914-2023. The bundled shaping font is composed by
mongfontbuilder from a source derived from Noto Sans Mongolian (OFL 1.1, outlines not included).
Notices in [NOTICE](https://github.com/Satsrag/mongol-norm/blob/main/NOTICE).

---

<a id="中文"></a>
## 中文

> [!WARNING]
> **Beta 版。** `shape` / `same_shape` 应该是稳定的。`normalize` 的输出目前是**按字形**编码的，**不是**按读音
> 书写的标准名义字符序列，后续版本有可能修改。例如 ᠮᠣᠩᠭᠣᠯ（`MA+O+ANG+GA+O+LA`）规范化后是 ᠮᠣᠠᠭ᠌ᠨ᠋ᠨ᠋ᠣᠯ
> （`MA+O+A+GA+FVS2+NA+FVS1+NA+FVS1+O+LA`）。如果要持久化规范化后的
> key，请同时保存 `canonical_version()`，版本变化时重建。

**mongol-norm** 判断两段传统蒙古文（回鹘式，Hudum）是否外形相同，并把它们映射成同一个 key。零依赖
Rust crate，另有 Python 包和命令行工具。

### 为什么

同一个看起来一样的蒙古文词，可以用很多种 Unicode 序列输入——A/E 在词中、词尾同形，O/U、OE/UE 共享字形，
FVS 和连接符（nirugu、ZWJ）也能用不同方式得到同一个字形。结果是：搜索找不到、去重失败、同一个词索引 key
不同。

![五种 sain 编码全部规范化为同一个形式](https://raw.githubusercontent.com/Satsrag/mongol-norm/main/assets/sain-variants.png)

mongol-norm 按 [UTN #57 v4](https://www.unicode.org/notes/tn57/tn57-4.html) 的整形规则（不需要字体）
算出字体会画出的**书写单元**：

- `shape(word)` —— 书写单元序列，即可见词形的指纹
- `same_shape(a, b)` —— 两种编码渲染结果是否相同
- `normalize(word)` / `normalize_text(text)` —— 同一个 shape 只输出一个 Unicode 字符串

用途：搜索和索引、去重、语料词频统计、拼写检查前的查词、清洗 OCR 或输入法输出。

### 在线体验

**[打开网页 →](https://www.satsrag.dev/norm/)** —— 输入蒙古文，实时查看 `shape`、`norm`、
Unicode 码点和规范化前后的竖排字形对照。使用 `hudum.otf` 显示，全部计算都在浏览器本地完成。

本地运行与构建说明见 [web/README.md](https://github.com/Satsrag/mongol-norm/blob/main/web/README.md)。

### 安装

**Rust**（MSRV 1.82，零依赖，可编译到 `wasm32-unknown-unknown`）：

```bash
cargo add mongol-norm
```

或写进 `Cargo.toml`：

```toml
[dependencies]
mongol-norm = "0.3.0"
```

**Python**（CPython ≥ 3.9，无运行时依赖，不需要 Rust 工具链）：

```bash
pip install mongol-norm
```

预编译 wheel 覆盖 Linux x86_64 / aarch64（glibc 与 musl）、macOS x86_64 / Apple silicon、Windows x64。
其他平台 pip 会从源码包构建，需要 Rust ≥ 1.83。

**命令行**——下面任意一种都会装上 `mongol-norm` 命令：

```bash
cargo install mongol-norm
pip install mongol-norm
```

### 用法

Rust：

```rust
use mongol_norm::{Error, Locale, Shaper};

fn main() -> Result<(), Error> {
    let shaper = Shaper::new(Locale::Mng);

    assert_eq!(shaper.shape_str("ᠰᠠᠢᠨ")?, "S+A+I+I+A");               // 整形
    assert!(shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")?);                      // 是否同形
    assert_eq!(shaper.normalize("ᠰᠡᠢᠨ")?, shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")?); // 规范化

    // 自由文本：只规范化蒙古文词，其余原样保留
    let text = shaper.normalize_text("Hello ᠰᠡᠢᠨ world")?;
    assert!(text.starts_with("Hello ") && text.ends_with(" world"));
    Ok(())
}
```

Python：

```python
from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")

shaper.shape("ᠰᠠᠢᠨ")                        # → ['S', 'A', 'I', 'I', 'A']
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")           # → True
shaper.normalize("ᠰᠡᠢᠨ")                    # → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")   # → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'

# 去重
{shaper.normalize(w) for w in ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]}   # → {'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}
```

命令行：

```bash
mongol-norm shape 'ᠰᠠᠢᠨ'                  # → S+A+I+I+A
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'            # 同形退出码 0，否则 1
mongol-norm normalize 'ᠰᠡᠢᠨ'              # 单个词
mongol-norm normalize-text -i in.txt -o out.txt
mongol-norm normalize --batch -i words.txt -o keys.txt   # 一行一个词
```

完整 API 见 [docs.rs](https://docs.rs/mongol-norm) 和
[Python 文档](https://github.com/Satsrag/mongol-norm/blob/main/python/README.pypi.md)；`mongol-norm --help`
列出全部参数。

### 说明

- 只支持 **回鹘式**（`MNG`），不支持托忒文、锡伯文、满文。
- 内置表无法编码的 shape 会报错而不是乱猜；`normalize_allow_fallback`（Python：`strict=False`）则原样返回。
- 整形 100% 通过 mongfontbuilder 的 `core-hud` 与 GB/T 25914-2023 `eac-hud` 测试集；规范化对全部语料词
  满足 `shape(normalize(x)) == shape(x)`。字体后端（Rust feature `font`，需要 Rust 1.85）用
  [harfrust](https://github.com/harfbuzz/harfrust)（HarfBuzz 的 Rust 移植）跑 mongfontbuilder 生成的
  回鹘式字体（去掉轮廓，30 KB），结果就是参照字体画出来的写法；通过所钉住的 mongfontbuilder 提交的回鹘式
  测试集，全部语料词与规则引擎一致（仅 `XIM11-46` 一处不同，上游也标为预期失败）。`normalize` 始终用规则引擎。
- 本项目用 [Claude Code](https://claude.ai/code) 编写，测试与核心代码经人工审核。遇到问题请提
  [issue](https://github.com/Satsrag/mongol-norm/issues)。

更多：[实现原理](https://github.com/Satsrag/mongol-norm/blob/main/docs/internals.md) ·
[数据格式](https://github.com/Satsrag/mongol-norm/blob/main/docs/data-format.md) ·
[开发与测试](https://github.com/Satsrag/mongol-norm/blob/main/docs/development.md) ·
[发布流程](https://github.com/Satsrag/mongol-norm/blob/main/docs/releasing.md)

### 许可

MIT，见 [LICENSE](https://github.com/Satsrag/mongol-norm/blob/main/LICENSE)。整形规则与数据源自
[UTN #57](https://www.unicode.org/notes/tn57/tn57-4.html) 和 Kushim Jiang 的
[mongfontbuilder](https://github.com/Kushim-Jiang/mongfontbuilder)（MIT）；测试集来自 mongfontbuilder 与
GB/T 25914-2023。内置整形字体由 mongfontbuilder 从 Noto Sans Mongolian 派生的源文件生成（OFL 1.1，不含轮廓）。
声明见 [NOTICE](https://github.com/Satsrag/mongol-norm/blob/main/NOTICE)。