unicode-lang 1.0.0

Unicode identifier rules (XID), normalization, char width.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

# unicode-lang v1.0.0 — API Freeze

**The surface is stable.** v0.2.0 delivered the three Unicode primitives; v1.0.0 freezes them. Everything the crate exposes — the eight functions, the `Form` enum, the `UNICODE_VERSION` constant, and the `std` / `alloc` / `serde` feature flags — is now stable under Semantic Versioning and will not change in a breaking way within the `1.x` series. No functional changes from 0.2.0; this release records the promise.

## What is unicode-lang?

The Unicode text primitives a language front end needs — identifier rules, display width, and all four normalization forms — from compact tables generated directly from the Unicode Character Database (16.0.0). The crate carries no third-party dependencies, is `no_std` (identifiers and width need no allocator), and is fully safe (`#![forbid(unsafe_code)]`). Every query is a branch-predictable binary search.

## The frozen surface

The whole public API, and all of it is stable as of 1.0.0:

- **Identifiers (UAX #31)** — `is_xid_start(char) -> bool`, `is_xid_continue(char) -> bool`, `is_xid(&str) -> bool`. Allocation-free.
- **Display width (UAX #11)** — `char_width(char) -> usize` and `str_width(&str) -> usize`, returning monospace column counts (0 / 1 / 2). Allocation-free.
- **Normalization (UAX #15)** — `normalize(&str, Form) -> String` and `is_normalized(&str, Form) -> bool`, selected by `Form` (`Nfc`, `Nfd`, `Nfkc`, `Nfkd`). Requires the `alloc` feature (default via `std`).
- **`UNICODE_VERSION`** — the `(u8, u8, u8)` UCD version the tables carry.
- **`serde`** (optional) — `Serialize` / `Deserialize` for `Form`.

## The SemVer promise

- Nothing in the frozen surface is removed or changed in a breaking way within `1.x`. A breaking change means a new major version.
- `1.x` releases may **add** to the surface (new functions, new trait impls) without breaking existing code.
- The **Unicode data version** behind `UNICODE_VERSION` may advance in a minor release; that can add assigned characters and refine width / normalization for previously-unassigned code points, and is not a breaking change to the API.
- The **serialized `serde` form** of `Form` is part of the contract and will not change within `1.x`.
- MSRV (Rust 1.85) is a compatibility surface: a raise is a minor, documented change, never a patch.

`docs/API.md` carries the same promise inline, per public item.

## Why freeze now

The surface is deliberately small — three focused capabilities, eight functions, one selector — and it is complete. Normalization passes the full official `NormalizationTest.txt` conformance suite; the identifier and width tables are generated directly from the UCD, so they are correct by construction. Every function is total (no fallible paths, no coercion surprises), the algebraic laws are property-tested across randomized inputs, and adding a helper later (for example a normalization iterator adaptor) is a non-breaking `1.x` change. There is nothing speculative left to design, so there is nothing to gain by staying pre-1.0.

## Performance

Unchanged from 0.2.0. Latest local Criterion means, Windows x86_64, Rust stable, release build:

| Operation                                | Time     |
|------------------------------------------|---------:|
| `is_xid_start` / `is_xid_continue`       |  ~3.2 ns |
| `is_xid` (11-scalar identifier)          |  ~37 ns  |
| `char_width` (per scalar)                |  ~5.2 ns |
| `str_width` (mixed 48-column string)     |  ~288 ns |
| `normalize` (ASCII, already normalized)  |  ~33 ns  |
| `normalize` (mixed scripts → NFC)        |  ~128 ns |
| `is_normalized` (ASCII → NFC)            |  ~13 ns  |

## Breaking changes

**None.** The 1.0.0 surface is identical to 0.2.0; this release only promises stability.

## Verification

Run on Windows x86_64 with testing on Linux (WSL2 Ubuntu), Rust stable; the same commands run in the CI matrix across Linux, macOS, and Windows:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
cargo build --no-default-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
```

All green. Counts at this tag:

- Default features: 23 unit + 8 property + 2 conformance + 9 doctests.
- `--all-features`: adds the `serde` round-trip test.
- The full `NormalizationTest.txt` suite (~19 965 records + whole-codespace identity) passes on Windows and Linux.

## Installation

```toml
[dependencies]
unicode-lang = "1"

# no_std without normalization (identifier + width only):
unicode-lang = { version = "1", default-features = false }

# With serde:
unicode-lang = { version = "1", features = ["serde"] }
```

MSRV: Rust 1.85.

## Documentation

- [README](https://github.com/jamesgober/unicode-lang/blob/main/README.md)
- [API Reference](https://github.com/jamesgober/unicode-lang/blob/main/docs/API.md)
- [CHANGELOG](https://github.com/jamesgober/unicode-lang/blob/main/CHANGELOG.md)

---

**Full diff:** [`v0.2.0...v1.0.0`](https://github.com/jamesgober/unicode-lang/compare/v0.2.0...v1.0.0).
**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/unicode-lang/blob/main/CHANGELOG.md#100---2026-07-01).