1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
//! SCRIPTS, and the one segmenter that cuts a text into runs of them.
//!
//! ```text
//! "Rust 2024 年鉴" ─▶ script_runs ─▶ [ latn "Rust 2024 " ] [ hani "年鉴" ]
//! ```
//!
//! A full-text index is one field with one tokenizer, and one tokenizer cannot serve text of every
//! writing system: whitespace splitting is right for Latin and useless for Han, and a dictionary
//! segmenter for Han is wrong for everything else. So a corpus that mixes them is FANNED — a run of
//! each script goes to the index built for it — and both ends of that arrangement need the same
//! answer to one question: *where does this text change script?*
//!
//! **ONE IMPLEMENTATION, BOTH SIDES.** The index side cuts a transcript into runs and feeds each to
//! the node that holds its script; the query side cuts the question the same way and asks only the
//! nodes it landed in. That the two agree is not a test that passes — it is one function, called
//! twice, which is the only construction under which *what was indexed is what is asked for* holds
//! by definition rather than by vigilance.
//!
//! # SEGMENTATION IS NOT TOKENISATION, and the line is the whole reason this is small
//!
//! What is here answers *which writing system* and nothing else. Where the WORDS are is a different
//! question with a different answer per script and per language, and it stays in the analyzer the
//! index is built with — jieba for Han, a dictionary for Thai, whitespace for Latin. This module
//! never splits a word, never lowercases, never folds a character, and has no opinion about
//! language: `en` and `fr` are one script and so are `zh` and the Han half of `ja`.
//!
//! # THE STANDARD IS INFRASTRUCTURE; THE POLICY IS DOWNSTREAM
//!
//! [`WritingSystem::of`] is UAX #24's question and [`script_runs`] is UAX #24's resolution, both stated
//! here once. What a deployment DOES with a run — which lane it routes to, which of them share an
//! index, what happens to a script nobody declared — is policy, and policy is written where the
//! declaration is. So nothing here mentions a lane, a tokenizer, an index or a backend, and the
//! module is reachable from a build that has none of them.
//!
//! # HAN IS UNIFIED, so a codepoint can only ever say [`Hani`](WritingSystem::Hani)
//!
//! Simplified and traditional Chinese share their codepoints — 中 is one character in both — so no
//! examination of a codepoint can tell them apart, and this module does not pretend to. That
//! distinction is a LANGUAGE fact carried by a language tag and folded inside an analyzer, one layer
//! away from here. The same unification puts Chinese and Japanese kanji in one script, which is why
//! a Japanese sentence cuts into Han, Hiragana and Katakana runs rather than into one — and why
//! putting those three back together is a router's job and not a segmenter's.
pub use ;
pub use WritingSystem;