unicode-lang 1.0.0

Performance First

Identifier and width queries are single-digit nanoseconds; normalization is dominated by its one output allocation and short-circuits when the input is already normalized. Latest local Criterion means (cargo bench --bench bench, Windows x86_64, Rust stable, release build):

Numbers vary by CPU and environment; run the suite on your target to establish a baseline. Data is generated from Unicode 16.0.0 (UNICODE_VERSION).

Features

• Identifier rules — is_xid_start, is_xid_continue, and the whole-string is_xid implement the UAX #31 XID profile a lexer needs.
• Normalization — all four forms (NFC, NFD, NFKC, NFKD) through one normalize, plus a fast is_normalized quick-check. Verified against the official conformance suite.
• Display width — char_width and str_width give monospace column counts (0 / 1 / 2), wcwidth-style.
• No dependencies — tables are generated from the UCD and embedded; nothing is pulled in at build time.
• no_std — identifier and width queries are allocation-free; normalization is gated behind alloc (on by default).
• Fully safe — #![forbid(unsafe_code)].
• Property-tested — algebraic laws (idempotence, form composition, width additivity) checked across randomized inputs with proptest.

Installation

[dependencies]
unicode-lang = "1"

# no_std without normalization (identifier + width only, no allocator):
unicode-lang = { version = "1", default-features = false }

# Serde support for the `Form` selector:
unicode-lang = { version = "1", features = ["serde"] }

MSRV is 1.85+ (Rust 2024 edition).

Quick Start

use unicode_lang::{char_width, is_xid, normalize, Form};

// Recognise an identifier that mixes scripts.
assert!(is_xid("Δpressure"));
assert!(!is_xid("1st"));      // cannot start with a digit

// Normalize: compose a base letter and a combining accent.
assert_eq!(normalize("e\u{0301}", Form::Nfc), "é");

// Compatibility folding: a ligature and a fullwidth digit.
assert_eq!(normalize("file", Form::Nfkc), "file");
assert_eq!(normalize("\u{FF11}", Form::Nfkc), "1");

// Measure display width for column alignment.
assert_eq!(char_width('A'), 1);
assert_eq!(char_width('世'), 2);   // wide CJK ideograph

Lexing identifiers

The two per-scalar predicates are what a hand-written lexer calls in its inner loop:

use unicode_lang::{is_xid_continue, is_xid_start};

fn lex_identifier(src: &str) -> Option<(&str, &str)> {
    let mut chars = src.char_indices();
    let (_, first) = chars.next()?;
    // Rust-style: an identifier may also start with '_'.
    if !(is_xid_start(first) || first == '_') {
        return None;
    }
    let end = chars
        .find(|&(_, c)| !is_xid_continue(c))
        .map_or(src.len(), |(i, _)| i);
    Some(src.split_at(end))
}

assert_eq!(lex_identifier("total99 = 1"), Some(("total99", " = 1")));
assert_eq!(lex_identifier("_private;"), Some(("_private", ";")));
assert_eq!(lex_identifier("1st"), None);

How it works

The public API is three small modules over one generated data file:

• Identifiers consult the XID_Start and XID_Continue ranges from DerivedCoreProperties. A query is one binary search over sorted, disjoint code-point ranges.
• Width classifies a scalar as zero-width (controls, combining marks, format characters, conjoining Hangul jamo), wide (East Asian Wide / Fullwidth), or the default one column.
• Normalization decomposes each scalar through the full canonical or compatibility mapping, reorders combining marks by canonical class, and — for NFC / NFKC — recomposes via the primary-composite table. Hangul is handled by formula. An already-normalized string is detected by the quick-check and returned untouched.

The tables are produced by dev/gen_tables.rs, a committed, dependency-free generator that reads the UCD text files and emits src/tables.rs. Regenerating against a newer Unicode release is a one-command step; the shipped crate contains only the generated data.

API Overview

For the complete reference with examples, see docs/API.md.

• is_xid_start / is_xid_continue / is_xid — UAX #31 identifier rules.
• char_width / str_width — monospace display width.
• normalize / is_normalized / Form — the four normalization forms.
• UNICODE_VERSION — the UCD version the tables were built from.

Feature Flags

Testing

cargo test                 # unit + property + doctests + curated conformance
cargo test --all-features  # adds the serde-gated paths
cargo bench --bench bench  # Criterion benchmarks

The property suite in tests/proptests.rs checks the algebraic laws — normalization is idempotent and stable, the forms compose as UAX #15 requires, and width is additive. tests/conformance.rs always runs a curated set of hard cases, and runs the entire official NormalizationTest.txt suite whenever the UCD data is present locally or in CI.

Cross-Platform Support

The crate is pure table lookups with no platform-specific code, so it behaves identically everywhere Rust runs. CI covers Linux, macOS, and Windows on both stable and the 1.85 MSRV; the full conformance suite is validated on Windows and Linux (WSL2 Ubuntu).

Contributing

See REPS.md for the engineering standards and the definition of done. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.