Skip to main content

fts5_cjk/
lib.rs

1//! Fixed CJK classification and compile-time transform profile metadata.
2//!
3//! [`is_cjk`] classifies one Rust Unicode scalar (`char`) using eight fixed,
4//! versioned ranges covering Han core, Han Extension A, Han Extension B,
5//! Hiragana, Katakana, Hangul Syllables, Hangul Jamo, and Hangul Compatibility
6//! Jamo. This is a compatibility policy, not an exhaustive or automatically
7//! updated definition of the latest Unicode CJK repertoire.
8//!
9//! The exact inclusive ranges are `U+4E00..U+9FFF`, `U+3400..U+4DBF`,
10//! `U+20000..U+2A6DF`, `U+3040..U+309F`, `U+30A0..U+30FF`,
11//! `U+AC00..U+D7A3`, `U+1100..U+11FF`, and `U+3130..U+318F`.
12//!
13//! [`ACTIVE_PROFILE`] is the complete identity of the contract compiled into
14//! this crate. The default build selects the NFC-off profile; the default-off
15//! `nfc` Cargo feature selects a distinct NFC-on profile and SQL UDF name.
16//! Cargo features are additive and may be unified by another dependency path,
17//! so consumers must inspect the compiled profile rather than infer it from
18//! deployment configuration.
19//!
20//! Persist and compare the complete profile: its ID, algorithm version, NFC
21//! state, SQL UDF name, and recommended tokenizer. Package semver and
22//! [`TRANSFORM_VERSION`] alone are insufficient because the NFC-off and NFC-on
23//! profiles share algorithm version 2 but are incompatible identities.
24//!
25//! ## Transform profile and semver policy
26//!
27//! Any change to transform output requires a new transform profile and a full
28//! reindex, regardless of the semver bump.
29//! Crate semver never signals index compatibility.
30//!
31//! Crate semver describes Rust API compatibility only; every release entry in
32//! the changelog states whether any profile changed.
33//!
34//! | Build | `id` | `algorithm_version` | `nfc` | `udf_name` | `recommended_tokenizer` |
35//! |-------|------|---------------------|-------|------------|-------------------------|
36//! | default (`nfc` feature off) | `fts5-cjk/v2;unicode61=6.1-default;nfc=off` | `2` | `false` | `fts5_cjk_v2` | `unicode61 remove_diacritics 2` |
37//! | `features = ["nfc"]` | `fts5-cjk/v2;unicode61=6.1-default;nfc=on` | `2` | `true` | `fts5_cjk_v2_nfc` | `unicode61 remove_diacritics 2` |
38//!
39//! docs.rs builds this crate with default features, so the constant values
40//! shown on docs.rs, such as [`UDF_NAME`], are the NFC-off profile's; the table
41//! above lists both profiles.
42//!
43//! Every output- or search-compatibility change requires a new profile
44//! identity. This includes classifier range changes, boundary-classification
45//! changes, NFC ordering, UDF semantics, literal escaping and phrase
46//! construction, and the tokenizer contract.
47//!
48//! If persisted index metadata differs from the compiled [`ACTIVE_PROFILE`],
49//! the consumer must refuse startup. Recovery requires an explicit rebuild or
50//! reindex with the compiled contract; only after that succeeds may the
51//! consumer stamp the new profile metadata. A consumer must never
52//! automatically restamp stale metadata, because that would falsely certify
53//! old index tokens as compatible.
54//!
55//! ## Pure transforms
56//!
57//! [`normalize_cjk`] applies the active profile's CJK token-boundary transform
58//! for both index and query input. It inserts one ASCII space between adjacent
59//! supported CJK scalars and at mixed boundaries where SQLite `unicode61`
60//! would otherwise retain one token. This includes the 25 combining marks
61//! consumed inside a token by the pinned `unicode61 remove_diacritics 2`
62//! tokenizer; it is not inferred from Unicode general categories alone.
63//! Existing punctuation and whitespace boundaries remain unchanged, and
64//! unchanged bytes use the `Cow::Borrowed` fast path. With the `nfc` feature,
65//! NFC composition happens before spacing.
66//!
67//! [`denormalize_cjk`] is a lossy, display-only helper. It collapses eligible
68//! CJK-CJK spacing but cannot distinguish original spaces or restore source
69//! text.
70//!
71//! ```
72//! use fts5_cjk::{denormalize_cjk, normalize_cjk};
73//!
74//! assert_eq!(normalize_cjk("AI개발중"), "AI 개 발 중");
75//! assert_eq!(normalize_cjk("개\u{0301}발"), "개 \u{0301} 발");
76//! assert_eq!(normalize_cjk("개\u{0345}발"), "개\u{0345}발");
77//! assert_eq!(denormalize_cjk("AI 개 발 중"), "AI 개발중");
78//! ```
79//!
80//! ## Literal query strings
81//!
82//! [`quote_fts5_phrase`] normalizes one complete raw value, doubles embedded
83//! double quotes, and adds exactly one outer quote pair. [`literal_query`]
84//! applies `split_whitespace` to raw input first, quotes each original term
85//! independently, preserves order, and joins phrases with one ASCII space.
86//!
87//! ```
88//! use fts5_cjk::{literal_query, quote_fts5_phrase};
89//!
90//! assert_eq!(quote_fts5_phrase("AI개"), "\"AI 개\"");
91//! assert_eq!(literal_query("개발 hello"), "\"개 발\" \"hello\"");
92//! assert_eq!(quote_fts5_phrase(""), "\"\"");
93//! assert_eq!(literal_query(" \t\n"), "");
94//! ```
95//!
96//! Consumers must reject an empty [`literal_query`] result before `MATCH`.
97//! These helpers accept literal values only; they do not implement Boolean,
98//! column, `NEAR`, prefix, negation, or parenthesized MATCH grammar.
99//!
100//! ## Consumer integration
101//!
102//! This crate has no SQLite dependency. The consumer owns its SQLite binding
103//! and registers the active transform as a scalar SQL function through it.
104//!
105//! ### Register the active transform
106//!
107//! ```rust,ignore
108//! // Consumer code, rusqlite 0.40 with the `functions` feature.
109//! // fts5-cjk itself has no SQLite dependency.
110//! use fts5_cjk::{ACTIVE_PROFILE, normalize_cjk};
111//! use rusqlite::{Connection, functions::FunctionFlags};
112//!
113//! fn register_fts5_cjk(conn: &Connection) -> rusqlite::Result<()> {
114//!     conn.create_scalar_function(
115//!         ACTIVE_PROFILE.udf_name,
116//!         1,
117//!         FunctionFlags::SQLITE_UTF8
118//!             | FunctionFlags::SQLITE_DETERMINISTIC
119//!             | FunctionFlags::SQLITE_INNOCUOUS,
120//!         |ctx| {
121//!             let raw: Option<String> = ctx.get(0)?;
122//!             Ok(raw.map(|raw| normalize_cjk(&raw).into_owned()))
123//!         },
124//!     )
125//! }
126//! ```
127//!
128//! Register exactly one text argument under `ACTIVE_PROFILE.udf_name` with the
129//! binding's UTF-8, deterministic, and `SQLITE_INNOCUOUS` flags, so that
130//! consumer-owned triggers and views may invoke it under trusted-schema
131//! hardening. Never use `SQLITE_DIRECTONLY`: index maintenance invokes the UDF
132//! from schema objects. If the binding cannot honor `SQLITE_INNOCUOUS`, keep
133//! `trusted_schema` enabled on those connections. SQL NULL maps to NULL, which
134//! FTS5 accepts, while any other non-text input errors. Propagate conversion and
135//! registration errors, return owned text, never copy the transform, and never
136//! derive the UDF name from package semver or deployment configuration.
137//!
138//! ### Keep index and query symmetric
139//!
140//! Every index write must reach an SQL object that invokes the registered
141//! active UDF, and the FTS5 table must use the exact
142//! `ACTIVE_PROFILE.recommended_tokenizer`. It must be an external-content
143//! table whose content source applies the UDF to raw source text; a contentful
144//! FTS5 table that stores UDF output is unsupported. Triggers, `'delete'`
145//! commands, and the content source read by `'rebuild'` must all supply
146//! UDF-normalized text. Never write the source table with
147//! `REPLACE` conflict resolution (`INSERT OR REPLACE`, `REPLACE INTO`,
148//! `UPDATE OR REPLACE`, or `ON CONFLICT REPLACE`): with SQLite's default
149//! `recursive_triggers = OFF` it deletes rows without firing DELETE triggers
150//! and leaves stale tokens. Write with UPSERT
151//! (`INSERT ... ON CONFLICT DO UPDATE`) instead. Query text must pass through
152//! [`literal_query`] or [`quote_fts5_phrase`] from the same compiled build.
153//! Bind the result as the `MATCH` value; never interpolate it into SQL text.
154//! Reject an empty [`literal_query`] result before `MATCH`.
155//!
156//! ### Rebuild and stamp
157//!
158//! Only an explicit rebuild transaction may certify index bytes:
159//!
160//! 1. Register the UDF under `ACTIVE_PROFILE.udf_name` before preparing SQL.
161//! 2. Begin a write transaction.
162//! 3. Create the FTS table with `ACTIVE_PROFILE.recommended_tokenizer`, as an
163//!    external-content table whose content source applies the UDF to raw
164//!    source text.
165//! 4. Run the FTS5 `'rebuild'` command for the table; it must read raw source
166//!    text through the UDF. A contentful FTS5 table that stores UDF output is
167//!    unsupported, because `'rebuild'` cannot reindex it.
168//! 5. Run a known positive probe and a known adjacency-negative probe.
169//! 6. Only after both pass, stamp all five profile fields and the schema identity.
170//! 7. Commit; roll back the whole transaction on any error.
171//!
172//! ### Read-only boot check
173//!
174//! 1. Register the UDF.
175//! 2. Read, and never write, the persisted profile and schema identity.
176//! 3. Compare every field exactly; missing metadata is a mismatch.
177//! 4. Run the same probes with read-only SQL.
178//! 5. Serve only if all checks pass; otherwise refuse before serving.
179//!
180//! Refusal diagnostics distinguish observed and expected fields and never
181//! include raw user query text. Boot never restamps; recovery is always the
182//! explicit rebuild above.
183//!
184//! ```
185//! use fts5_cjk::ACTIVE_PROFILE;
186//!
187//! // Owned copy of the five fields as read back from consumer storage.
188//! type Stored = (String, u32, bool, String, String);
189//!
190//! fn stamp() -> Stored {
191//!     let p = ACTIVE_PROFILE;
192//!     (p.id.into(), p.algorithm_version, p.nfc, p.udf_name.into(), p.recommended_tokenizer.into())
193//! }
194//!
195//! // `None` is missing metadata: a mismatch, never a state for boot to fill.
196//! fn boot_allowed(stored: Option<&Stored>) -> bool {
197//!     stored == Some(&stamp())
198//! }
199//!
200//! let stored = stamp();
201//! assert!(boot_allowed(Some(&stored)));
202//! let flipped = (stored.0.clone(), stored.1, !stored.2, stored.3.clone(), stored.4.clone());
203//! assert!(!boot_allowed(Some(&flipped)));
204//! assert!(!boot_allowed(None));
205//! ```
206//!
207//! ## Real FTS5 evidence
208//!
209//! The repository integration target executes both public helper outputs as
210//! bound MATCH values in one direct SQLite process and records its version and
211//! source identity. Both explicit profiles pass a fixed 10-case
212//! recall/precision matrix plus two Hangul canonical-direction cases, including
213//! mixed-boundary and embedded-quote positives and separated/reversed
214//! adjacency negatives. The Hangul directions match under `nfc`; default
215//! evidence is limited to the observed distinction for the pinned `가`/`가`
216//! fixture and makes no general canonical-equivalence or
217//! canonical-inequivalence claim.
218//!
219//! ## Scope and limits
220//!
221//! - The fixed eight ranges above operate on UTF-8 decoded Rust Unicode
222//!   scalars; they are not an automatically expanding Unicode repertoire.
223//! - Search is substring-like only within an ordered, adjacent run of supported
224//!   CJK scalars. It is not morphology, stemming, transliteration, dictionary
225//!   lookup, or a general substring search engine.
226//! - Spacing at mixed CJK/token-character boundaries is intentional and
227//!   remains visible after [`denormalize_cjk`]. Denormalization removes
228//!   eligible CJK-CJK spaces only and is lossy, display-only output handling.
229//! - The `nfc` feature is default-off and composes before spacing. It selects a
230//!   distinct complete profile, so index and query paths must use the same
231//!   compiled profile; default makes no general canonical claim.
232//! - The helpers implement literal MATCH grammar only. An empty
233//!   [`literal_query`] result is a rejection signal, not an empty search.
234//! - The consumer owns its SQLite binding, UDF registration, connection
235//!   ordering, schema, migrations, ranking, snippets, and runtime profile
236//!   selection.
237//!
238//! The README repeats this contract with a full persistence example. This
239//! crate implements no consumer schema or migration.
240
241#![forbid(unsafe_code)]
242
243mod classification;
244mod literal;
245mod profile;
246mod transform;
247
248pub use classification::is_cjk;
249pub use literal::{literal_query, quote_fts5_phrase};
250pub use profile::{ACTIVE_PROFILE, TRANSFORM_VERSION, TransformProfile, UDF_NAME};
251pub use transform::{denormalize_cjk, normalize_cjk};
252
253#[cfg(doctest)]
254#[doc = include_str!("../README.md")]
255struct ReadmeDoctests;