unicode-lang 1.0.0

Unicode identifier rules (XID), normalization, char width.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154

# unicode-lang v0.2.0 — The Core

**The three primitives the crate exists for.** v0.2.0 lands Unicode identifier rules, display width, and all four normalization forms — a zero-dependency, `no_std` surface backed by compact tables generated from the Unicode Character Database (16.0.0). Normalization is verified against the official conformance suite. This is the hard part of the roadmap, and it is not deferred.

## What is unicode-lang?

The Unicode text primitives a language front end needs. A lexer has to answer three questions the standard library does not: *may this scalar start or continue an identifier?* (UAX #31), *are these two strings the same once normalized?* (UAX #15), and *how many columns does this text occupy?* (UAX #11). unicode-lang answers all three from sorted lookup tables generated directly from the UCD, so the crate carries no third-party dependencies and every query is a branch-predictable binary search.

## What's new in 0.2.0

### Identifiers — `is_xid_start`, `is_xid_continue`, `is_xid`

The `XID_Start` / `XID_Continue` derived properties, plus a whole-string convenience. These are what a hand-written lexer calls in its inner loop: the first scalar of an identifier must be `XID_Start`, each subsequent scalar `XID_Continue`. The `XID` forms are closure-stable — normalizing a valid identifier never makes it invalid.

```rust
use unicode_lang::{is_xid, is_xid_continue, is_xid_start};

assert!(is_xid("Δpressure"));
assert!(!is_xid("1st"));           // cannot start with a digit
assert!(!is_xid("_name"));         // underscore is XID_Continue, not XID_Start

assert!(is_xid_start('日'));
assert!(is_xid_continue('_'));     // connector punctuation continues
```

Both predicates are allocation-free and available with no features enabled.

### Display width — `char_width`, `str_width`

Monospace column counts (0 / 1 / 2), `wcwidth`-style: zero for controls, combining marks, and format characters; two for East Asian Wide and Fullwidth; one for everything else. East Asian *Ambiguous* is treated as narrow.

```rust
use unicode_lang::{char_width, str_width};

assert_eq!(char_width('A'), 1);
assert_eq!(char_width('世'), 2);        // wide CJK ideograph
assert_eq!(char_width('\u{0301}'), 0);  // COMBINING ACUTE ACCENT

assert_eq!(str_width("日本語"), 6);
assert_eq!(str_width("e\u{0301}"), 1);  // base + mark advances one column
```

### Normalization — all four forms

`normalize(&str, Form)` and `is_normalized(&str, Form)`, selected by the new `Form` enum (`Nfc`, `Nfd`, `Nfkc`, `Nfkd`). The implementation runs the standard UAX #15 pipeline: full canonical or compatibility decomposition, a stable sort of combining marks by canonical class, and — for the composing forms — recomposition through the primary-composite table. Hangul is handled by formula.

```rust
use unicode_lang::{normalize, Form};

// Canonical composition and decomposition.
assert_eq!(normalize("e\u{0301}", Form::Nfc), "é");
assert_eq!(normalize("é", Form::Nfd), "e\u{0301}");

// Compatibility folding.
assert_eq!(normalize("file", Form::Nfkc), "file");
assert_eq!(normalize("\u{FF21}", Form::Nfkc), "A");
```

`normalize` short-circuits with the Unicode quick-check: text that is already in the requested form is returned after one allocation-free scan. `is_normalized` decides most inputs from the quick-check alone and never allocates a `String`, so a caller can skip normalization entirely on the common path.

```rust
use unicode_lang::{is_normalized, Form};

assert!(is_normalized("é", Form::Nfc));
assert!(!is_normalized("e\u{0301}", Form::Nfc));
assert!(is_normalized("e\u{0301}", Form::Nfd));
```

This API requires the `alloc` feature, enabled by default through `std`.

### Zero dependencies, generated from the UCD

There are no third-party runtime dependencies. The lookup tables in `src/tables.rs` are produced by `dev/gen_tables.rs`, a committed, dependency-free generator that reads the UCD text files (`UnicodeData`, `DerivedCoreProperties`, `DerivedNormalizationProps`, `EastAsianWidth`) and emits the sorted range and decomposition tables the crate binary-searches. The shipped crate contains only the generated data; `UNICODE_VERSION` records the release it was built from.

### `no_std` and feature layout

The crate is `no_std`. Identifier and width queries need no allocator and are available with no features at all. Normalization allocates its output, so it is gated behind `alloc` (implied by the default `std` feature). The optional `serde` feature derives `Serialize` / `Deserialize` for `Form`.

### Tests and benchmarks

- A `proptest` suite (`tests/proptests.rs`) asserts the algebraic laws: normalization is idempotent and stable, `is_normalized` agrees with `normalize`, the forms compose as UAX #15 requires (`NFD(NFC(s)) == NFD(s)`), and width is additive.
- A conformance suite (`tests/conformance.rs`) runs the entire official `NormalizationTest.txt` when the UCD data is present — every one of its ~19 000 vectors plus the whole-codespace identity rule — and a curated set of hard cases unconditionally.
- Criterion benchmarks (`benches/bench.rs`) cover the identifier predicates, width, and all four normalization forms across representative inputs.

## Performance

Latest local Criterion means, Windows x86_64, Rust stable, release build:

| Operation                                | Time     |
|------------------------------------------|---------:|
| `is_xid_start` / `is_xid_continue`       |  ~3.2 ns |
| `is_xid` (11-scalar identifier)          |  ~37 ns  |
| `char_width` (per scalar)                |  ~5.2 ns |
| `str_width` (mixed 48-column string)     |  ~288 ns |
| `normalize` (ASCII, already normalized)  |  ~33 ns  |
| `normalize` (mixed scripts → NFC)        |  ~128 ns |
| `is_normalized` (ASCII → NFC)            |  ~13 ns  |

Numbers vary by CPU; run `cargo bench` on your target for a baseline.

## Breaking changes

**None** that affect real code — v0.1.0 was a scaffold with no public API. Two structural repairs landed on the way: the manifest `keywords` / `categories` were unquoted bare identifiers (the crate did not parse as valid TOML) and are now proper string arrays, and the `clippy.toml` MSRV was aligned with `Cargo.toml` at 1.85.

## Verification

Run on Windows x86_64 with testing on Linux (WSL2 Ubuntu), Rust stable; the same commands run in the CI matrix across Linux, macOS, and Windows:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
cargo build --no-default-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
```

All green. Counts at this tag:

- Default features: 23 unit + 8 property + 2 conformance + 9 doctests.
- `--all-features`: adds the `serde` round-trip test and its gated paths.
- The full `NormalizationTest.txt` suite (~19 965 records + whole-codespace identity) passes on Windows and Linux.

## What's next

- **1.0.0 — API freeze.** Mark `docs/API.md` stable, record the SemVer promise, and confirm the full test and benchmark suite green on all three platforms.

## Installation

```toml
[dependencies]
unicode-lang = "0.2"

# no_std without normalization (identifier + width only):
unicode-lang = { version = "0.2", default-features = false }

# With serde:
unicode-lang = { version = "0.2", features = ["serde"] }
```

MSRV: Rust 1.85.

## Documentation

- [README](https://github.com/jamesgober/unicode-lang/blob/main/README.md)
- [API Reference](https://github.com/jamesgober/unicode-lang/blob/main/docs/API.md)
- [CHANGELOG](https://github.com/jamesgober/unicode-lang/blob/main/CHANGELOG.md)

---

**Full diff:** [`v0.1.0...v0.2.0`](https://github.com/jamesgober/unicode-lang/compare/v0.1.0...v0.2.0).
**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/unicode-lang/blob/main/CHANGELOG.md#020---2026-07-01).