uqa-analysis

uqa-analysis is the UQA Engine crate for tokenizers, character filters, token filters, and analyzers.

The optional nori feature exposes a validated immutable Korean dictionary, user-rule compiler, and native rolling Viterbi tokenizer with lossless UTF-16 morphology and graph attributes. nori-tools adds the offline packer and complete neutral-model verifier. The standalone tokenizer, default Korean analyzer, POS/reading/simple-lowercase filters, optional exact-decimal number composition, and separate normalization are verified against pinned Lucene fixtures. The native common-token bridge preserves raw terms, morphology, graph/end state, and corrected source spans through generic filters. Korean filters share one implementation across native and common tokens, including absent morphology, retained source provenance, and exact composed spans. NoriResources resolves immutable bundles by exact artifact hash, shares bounded dictionary and user-rule caches, and uses the bundled data through the optional nori feature without implicit I/O. Analyzer::compile() prepares immutable existing pipelines with fixed expressions, stop sets, and synonym maps while uncompiled APIs keep file reload behavior. AnalyzerDescriptor snapshots canonical resolved inputs and runtime profiles under a versioned fingerprint; AnalyzerResources restores those snapshots and shares compiled revisions under bounded cache ownership. Korean tokenizer/filter configurations now compile through the common pipeline with exact resource identities, user-rule snapshots, normalization, and bounded shared ownership. Memory, SQLite, and redb retain exact durable analyzer revisions and complete token graphs; public SQL and bindings execute graph phrases and highlight original source spans. Ported code retains the Lucene license, notice, and modification attribution in THIRD-PARTY/. See the analyzer reference and bundle format.

The independent optional kuromoji feature exposes KuromojiDictionary and the bundled Japanese dictionary from uqa-kuromoji-data. It validates the complete immutable model, including all six Japanese morphology attributes, unknown classes, connection costs, Java Unicode values, default stop sets and completion mappings. KuromojiResources resolves exact immutable bundles and interns Japanese user rules with bounded retention; its host resolver has no implicit filesystem or network fallback. AnalyzerResources::builder can install both typed language resource owners together, and the existing Nori constructor remains available. kuromoji-tools provides an offline packer and exhaustive comparison with the pinned Docker export. Generic and Nori configurations do not acquire the Japanese data dependency unless kuromoji is enabled. Native JapaneseTokenizer exposes NORMAL/SEARCH/EXTENDED modes, punctuation/compound choices, six morphology attributes and exact graph/end state with bounded resegmentation, cancellation and retained memory. All 209 pinned tokenizer cases match. The common token bridge shares source correction, graph validation and allocation transfer with Nori. JapaneseMorphology preserves all six optional attributes through generic token filters, and tokenize_mapped_budgeted retains original spans after character filtering. Native N-best cost/example selection matches 98 additional Docker cases, including signed costs, stable alternative graphs, lazy attributes and preparation errors. Byte/count/work allowances and cancellation cover its independent scratch; non-positive costs allocate no alternative arrays. Standalone JapaneseAnalyzer and its base-form, exact POS and word stops, Katakana stemmer and simple-lowercase filters, plus optional Hiragana/Katakana small-kana expansion and kana/romaji reading conversion, match 195 additional Docker cases including every UTF-16 unit and the complete Katakana reading context matrix. Native/common streams share one filter algorithm and retain source, graph and terminal attributes. Reading conversion preserves absent/empty attributes and uses bounded modified-Hepburn lookahead separately from completion romanization. Optional JapaneseFilter::Number and the scalar/raw UTF-16 number normalizers use shared exact decimals, prefix parsing and retained lookahead streams with Japanese numeral and attribute policies. Their separate 220-case Docker corpus verifies exact numeric grammar, all UTF-16 units in two contexts, graph/terminal propagation, long coefficients and source spans; bounded outputs, cancellation and retained leases are tested on both native and common tokens. Custom chains preserve lazy user-field access errors, and normalization applies only width and simple lowercase. Lookup preparation and runtime mutation have explicit memory/count limits and cancellation. CharFilter::KuromojiIterationMark adds independently configurable kanji/kana expansion with exact original-source span and voicing behavior, verified by 189 Docker text/offset cases including every BMP scalar with all five marks under all four flag settings. It uses the common reserved character-edit owner and participates in immutable generic pipeline descriptors. Ordered completion romanization now matches 396 Docker cases through the existing lexical-rank owner, with complete-product counts, retained result buffers and explicit work limits. Completion INDEX/QUERY streams and a dedicated analyzer constructor now bring the complete corpus to 522 cases. They reset graph/keyword/morphology state through owning token adapters, retain terminal/source behavior and select width-only normalization separately from the lowercase analysis chain. Native generated origins and common morphology are absent. Explicit normalization configurations now compile and restore with typed exact profile hashes, separate width-only or width-plus-simple-lowercase plans, strict properties and unchanged legacy omission. Native and compiled normalization share one reserved scalar conversion owner; dictionary aliases resolve once per language pipeline and retained handles preserve their original profiles. Japanese tokenizer configurations now compile through common pipelines and restore exact dictionary/user snapshots and effective N-best costs without repeating example probes. Native/common attribute failures remain deferred through filters, and linear storage adapters reject Japanese graphs. Common simple-lowercase configuration now accepts explicit typed language profiles while preserving legacy Nori strings. It reuses the owning native filters and the shared resource snapshot; compiled execution retains prepared filters without resource lookup. Six Japanese base-form/stem/kana/reading/number filter configurations now compile and restore without dictionary lookup or retention. Their existing private native/common interface accepts an absent model for these rules and still requires one for defaults, Unicode and completion. POS/word-stop/completion configurations and both built-ins also compile and restore. Default stop sets freeze as sorted original strings; expansion-only models are released, while case-insensitive stops and completion retain exact profiles. Ordinary normalization performs width plus simple lowercase; completion normalization performs width only. Provider/SQL and actual Python/Node.js/WASM integration are verified; the bundle format describes this dictionary API.

kuromoji::UserDictionary compiles Japanese CSV against a selected immutable model, preserving exact source identity, phrase/word order, readings/POS, segmentation and overlapping longest lookup. It shares preparation limits and lexical traversal with Nori while retaining Japanese parsing and duplicate errors. Its public API and executed example are in the kuromoji module; 62 pinned Docker cases verify complete user-rule outputs and failures independently of the Japanese tokenizer. See the standalone Japanese tokenization reference for the native runtime API.

Applications should depend on uqa-engine or uqa-client. See the repository README and the manual.