uqa-analysis

uqa-analysis is the UQA Engine crate for tokenizers, character filters, token filters, and analyzers.

The optional nori feature exposes a validated immutable Korean dictionary, user-rule compiler, and native rolling Viterbi tokenizer with lossless UTF-16 morphology and graph attributes. nori-tools adds the offline packer and complete neutral-model verifier. The standalone tokenizer, default Korean analyzer, POS/reading/simple-lowercase filters, optional exact-decimal number composition, and separate normalization are verified against pinned Lucene fixtures. The native common-token bridge preserves raw terms, morphology, graph/end state, and corrected source spans through generic filters. Korean filters share one implementation across native and common tokens, including absent morphology, retained source provenance, and exact composed spans. NoriResources resolves immutable bundles by exact artifact hash, shares bounded dictionary and user-rule caches, and uses the bundled data through the optional nori feature without implicit I/O. Analyzer::compile() prepares immutable existing pipelines with fixed expressions, stop sets, and synonym maps while uncompiled APIs keep file reload behavior. AnalyzerDescriptor snapshots canonical resolved inputs and runtime profiles under a versioned fingerprint; AnalyzerResources restores those snapshots and shares compiled revisions under bounded cache ownership. Korean tokenizer/filter configurations now compile through the common pipeline with exact resource identities, user-rule snapshots, normalization, and bounded shared ownership. Memory, SQLite, and redb retain exact durable analyzer revisions and complete token graphs; public SQL and bindings execute graph phrases and highlight original source spans. Ported code retains the Lucene license, notice, and modification attribution in THIRD-PARTY/. See the analyzer reference and bundle format.

Applications should depend on uqa-engine or uqa-client. See the repository README and the manual.