`vwi-core` — Vietnamese Writing Intelligence Core Engine

vwi-core is the pure-Rust linguistic engine powering Vietnamese Writing Intelligence. It combines a statistical Viterbi Trellis word segmenter (72,564 compound dictionary), Universal Dependencies (UPOS) part-of-speech tagger, and a functional Topic-Comment shallow parser inspired by Prof. Cao Xuân Hạo's framework.

Key Features

Viterbi Trellis Word Segmentation: Disambiguates overlapping Vietnamese compounds using Unigram/Bigram log-probabilities and Double-Array Aho-Corasick (daachorse).
Universal Dependencies POS Tagging: $O(\log N)$ binary search lexicon with $13 \times 13$ transition matrix and morphological OOV heuristics.
Topic-Comment Shallow Parser: Detects Crippled Prepositional Topics (Khung đề què) and Preposition-Conjunction Conflicts without fragile token-limit regexes.
25+ Linguistic & Stylistic Rules: Covers artificial passives, translationese, pleonasms, concessive mood misuse, bureaucratic verb padding, and statistically mined AI clichés ($G^2$ Log-Likelihood).
Parallel Batch Auditing: Multi-threaded document auditing via rayon with configurable rule overrides and non-linear 0–100 quality scoring (AuditReport).

Quick Start

use vwi_core::{Auditor, tokenize_viterbi};

fn main() {
    let auditor = Auditor::default();
    let report = auditor.audit_document(
        "Trong tình hình kinh tế hiện nay đòi hỏi doanh nghiệp phải tái cơ cấu toàn diện."
    );
    println!("Quality Score: {}/100 ({})", report.score, report.grade);
    for f in &report.findings {
        println!("[{}] {}: {}", f.severity, f.pattern_id, f.matched_span);
    }

    let tokens = tokenize_viterbi("Học sinh học sinh học tại trường.");
    for t in tokens {
        println!("{} -> {:?}", t.text, t.pos);
    }
}

Scope & Current Limitations

What `vwi-core` does well:

High-Speed Deterministic Linter: Zero-allocation matching, embedded 72.5k lexicon with Viterbi dynamic programming. Sub-millisecond latency.
Functional Topic-Comment Grammar: Catches crippled topics ("Trong bối cảnh... đòi hỏi"), artificial passives ("được... bởi"), and translationese.
AuditReport Quality Gate: Non-linear error density scoring (0–100) per 200 effective words for CI/CD integration.

What is OUT OF SCOPE for `vwi-core`:

Not a Deep Learning Transformer: Does not replace PhoBERT or LLMs for deep semantic representation, summarization, or open-domain Q&A.
POS Tagger Boundaries: Bigram HMM + morphological heuristics. Optimized for speed and low memory; edge cases in ancient poetry or non-standard slang fall back to single-syllable tags.
No Social Media Teencode Normalization: Tailored for standard Vietnamese (journalism, administrative, literature).
Shallow Parser, not Full Dependency Treebank: Focuses on essential functional components (topic-comment frames, predicates, connectors) rather than full parse trees.

License

Distributed under the MIT License.

vwi-core 0.3.1

vwi-core — Vietnamese Writing Intelligence Core Engine