unicode_lang/lib.rs
1//! # unicode_lang
2//!
3//! The Unicode text primitives a language front end needs: identifier rules,
4//! normalization, and display width — with correct-by-construction tables and
5//! no third-party dependencies.
6//!
7//! A lexer has to answer three Unicode questions that the standard library does
8//! not: *may this scalar start or continue an identifier?* (UAX #31), *are
9//! these two strings the same once normalized?* (UAX #15), and *how many
10//! columns does this text occupy?* (UAX #11). `unicode-lang` answers all three
11//! from compact lookup tables generated directly from the Unicode Character
12//! Database, so the crate carries no runtime dependencies and every query is a
13//! branch-predictable binary search.
14//!
15//! ## At a glance
16//!
17//! - **Identifiers** — [`is_xid_start`], [`is_xid_continue`], and the
18//! whole-string [`is_xid`] implement the UAX #31 `XID` profile.
19//! - **Width** — [`char_width`] and [`str_width`] give monospace column counts
20//! (0 / 1 / 2), `wcwidth`-style.
21//! - **Normalization** — [`normalize`] and [`is_normalized`], selected by
22//! [`Form`], implement all four forms (NFC, NFD, NFKC, NFKD) and are verified
23//! against the official conformance suite. *(Requires the `alloc` feature,
24//! enabled by default via `std`.)*
25//! - [`UNICODE_VERSION`] records the UCD version the tables were built from.
26//!
27//! ## Example
28//!
29//! ```
30//! use unicode_lang::{char_width, is_xid, normalize, Form};
31//!
32//! // Recognise an identifier that mixes scripts.
33//! assert!(is_xid("Δpressure"));
34//!
35//! // Fold a compatibility ligature and recompose.
36//! assert_eq!(normalize("file", Form::Nfkc), "file");
37//!
38//! // Measure a mixed-width string for column alignment.
39//! assert_eq!(char_width('世'), 2);
40//! ```
41//!
42//! ## `no_std`
43//!
44//! The crate is `no_std`. Identifier and width queries need no allocator and
45//! are available with no features at all. Normalization allocates its output,
46//! so it is gated behind the `alloc` feature (implied by the default `std`
47//! feature); disable default features for a bare `no_std` build without
48//! normalization. The optional `serde` feature derives `Serialize` /
49//! `Deserialize` for [`Form`].
50
51#![cfg_attr(not(feature = "std"), no_std)]
52#![cfg_attr(docsrs, feature(doc_cfg))]
53#![forbid(unsafe_code)]
54#![deny(missing_docs)]
55#![deny(unused_must_use)]
56#![deny(unused_results)]
57#![deny(clippy::unwrap_used)]
58#![deny(clippy::expect_used)]
59#![deny(clippy::todo)]
60#![deny(clippy::unimplemented)]
61#![deny(clippy::print_stdout)]
62#![deny(clippy::print_stderr)]
63#![deny(clippy::dbg_macro)]
64#![deny(clippy::unreachable)]
65
66mod ident;
67mod lookup;
68mod tables;
69mod width;
70
71#[cfg(feature = "alloc")]
72mod normalize;
73
74pub use ident::{is_xid, is_xid_continue, is_xid_start};
75pub use tables::UNICODE_VERSION;
76pub use width::{char_width, str_width};
77
78#[cfg(feature = "alloc")]
79pub use normalize::{Form, is_normalized, normalize};