# uqa-analysis
`uqa-analysis` is the UQA Engine crate for tokenizers, character filters, token filters, and analyzers.
The optional `nori` feature exposes a validated immutable Korean dictionary, user-rule compiler, and native rolling Viterbi tokenizer with lossless UTF-16 morphology and graph attributes. `nori-tools` adds the offline packer and complete neutral-model verifier. The standalone tokenizer, default Korean analyzer, POS/reading/simple-lowercase filters, optional exact-decimal number composition, and separate normalization are verified against pinned Lucene fixtures. The native common-token bridge preserves raw terms, morphology, graph/end state, and corrected source spans through generic filters. Korean filters share one implementation across native and common tokens, including absent morphology, retained source provenance, and exact composed spans. `NoriResources` resolves immutable bundles by exact artifact hash, shares bounded dictionary and user-rule caches, and uses the bundled data through the optional `nori` feature without implicit I/O. `Analyzer::compile()` prepares immutable existing pipelines with fixed expressions, stop sets, and synonym maps while uncompiled APIs keep file reload behavior. `AnalyzerDescriptor` snapshots canonical resolved inputs and runtime profiles under a versioned fingerprint; `AnalyzerResources` restores those snapshots and shares compiled revisions under bounded cache ownership. Korean tokenizer/filter configurations now compile through the common pipeline with exact resource identities, user-rule snapshots, normalization, and bounded shared ownership. Memory, SQLite, and redb retain exact durable analyzer revisions and complete token graphs; public SQL and bindings execute graph phrases and highlight original source spans. Ported code retains the Lucene license, notice, and modification attribution in `THIRD-PARTY/`. See the [analyzer reference](https://github.com/cognica-io/uqa-engine/blob/main/docs/manual/reference/06-text-analyzers.md#standalone-korean-tokenization) and [bundle format](https://github.com/cognica-io/uqa-engine/blob/main/docs/design/nori-bundle-format.md).
The independent optional `kuromoji` feature exposes `KuromojiDictionary` and the bundled Japanese dictionary from `uqa-kuromoji-data`. It validates the complete immutable model, including all six Japanese morphology attributes, unknown classes, connection costs, Java Unicode values, default stop sets and completion mappings. `KuromojiResources` resolves exact immutable bundles and interns Japanese user rules with bounded retention; its host resolver has no implicit filesystem or network fallback. `AnalyzerResources::builder` can install both typed language resource owners together, and the existing Nori constructor remains available. `kuromoji-tools` provides an offline packer and exhaustive comparison with the pinned Docker export. Generic and Nori configurations do not acquire the Japanese data dependency unless `kuromoji` is enabled. Native `JapaneseTokenizer` exposes NORMAL/SEARCH/EXTENDED modes, punctuation/compound choices, six morphology attributes and exact graph/end state with bounded resegmentation, cancellation and retained memory. All 209 pinned tokenizer cases match. The common token bridge shares source correction, graph validation and allocation transfer with Nori. `JapaneseMorphology` preserves all six optional attributes through generic token filters, and `tokenize_mapped_budgeted` retains original spans after character filtering. Native N-best cost/example selection matches 98 additional Docker cases, including signed costs, stable alternative graphs, lazy attributes and preparation errors. Byte/count/work allowances and cancellation cover its independent scratch; non-positive costs allocate no alternative arrays. Standalone `JapaneseAnalyzer` and its base-form, exact POS and word stops, Katakana stemmer and simple-lowercase filters, plus optional Hiragana/Katakana small-kana expansion and kana/romaji reading conversion, match 195 additional Docker cases including every UTF-16 unit and the complete Katakana reading context matrix. Native/common streams share one filter algorithm and retain source, graph and terminal attributes. Reading conversion preserves absent/empty attributes and uses bounded modified-Hepburn lookahead separately from completion romanization. Optional `JapaneseFilter::Number` and the scalar/raw UTF-16 number normalizers use shared exact decimals, prefix parsing and retained lookahead streams with Japanese numeral and attribute policies. Their separate 220-case Docker corpus verifies exact numeric grammar, all UTF-16 units in two contexts, graph/terminal propagation, long coefficients and source spans; bounded outputs, cancellation and retained leases are tested on both native and common tokens. Custom chains preserve lazy user-field access errors, and normalization applies only width and simple lowercase. Lookup preparation and runtime mutation have explicit memory/count limits and cancellation. `CharFilter::KuromojiIterationMark` adds independently configurable kanji/kana expansion with exact original-source span and voicing behavior, verified by 189 Docker text/offset cases including every BMP scalar with all five marks under all four flag settings. It uses the common reserved character-edit owner and participates in immutable generic pipeline descriptors. Ordered completion romanization now matches 396 Docker cases through the existing lexical-rank owner, with complete-product counts, retained result buffers and explicit work limits. Completion INDEX/QUERY streams and a dedicated analyzer constructor now bring the complete corpus to 522 cases. They reset graph/keyword/morphology state through owning token adapters, retain terminal/source behavior and select width-only normalization separately from the lowercase analysis chain. Native generated origins and common morphology are absent. Explicit normalization configurations now compile and restore with typed exact profile hashes, separate width-only or width-plus-simple-lowercase plans, strict properties and unchanged legacy omission. Native and compiled normalization share one reserved scalar conversion owner; dictionary aliases resolve once per language pipeline and retained handles preserve their original profiles. Japanese tokenizer configurations now compile through common pipelines and restore exact dictionary/user snapshots and effective N-best costs without repeating example probes. Native/common attribute failures remain deferred through filters, and linear storage adapters reject Japanese graphs. Common simple-lowercase configuration now accepts explicit typed language profiles while preserving legacy Nori strings. It reuses the owning native filters and the shared resource snapshot; compiled execution retains prepared filters without resource lookup. Six Japanese base-form/stem/kana/reading/number filter configurations now compile and restore without dictionary lookup or retention. Their existing private native/common interface accepts an absent model for these rules and still requires one for defaults, Unicode and completion. POS/word-stop/completion configurations and both built-ins also compile and restore. Default stop sets freeze as sorted original strings; expansion-only models are released, while case-insensitive stops and completion retain exact profiles. Ordinary normalization performs width plus simple lowercase; completion normalization performs width only. Provider/SQL and actual Python/Node.js/WASM integration are verified; the [bundle format](https://github.com/cognica-io/uqa-engine/blob/main/docs/design/kuromoji-bundle-format.md) describes this dictionary API.
`kuromoji::UserDictionary` compiles Japanese CSV against a selected immutable model, preserving exact source identity, phrase/word order, readings/POS, segmentation and overlapping longest lookup. It shares preparation limits and lexical traversal with Nori while retaining Japanese parsing and duplicate errors. Its public API and executed example are in the `kuromoji` module; 62 pinned Docker cases verify complete user-rule outputs and failures independently of the Japanese tokenizer. See the [standalone Japanese tokenization reference](https://github.com/cognica-io/uqa-engine/blob/main/docs/manual/reference/06-text-analyzers.md#standalone-japanese-tokenization) for the native runtime API.
Applications should depend on `uqa-engine` or `uqa-client`. See the [repository README](https://github.com/cognica-io/uqa-engine) and the [manual](https://github.com/cognica-io/uqa-engine/blob/main/docs/manual/README.md).