Skip to main content

Crate delarocha

Crate delarocha 

Source
Expand description

Japanese morphological analyzer that reads MeCab-format system dictionaries (such as IPADIC) and produces MeCab/Vibrato-compatible segmentation.

  • ffi (feature zig-ffi): the fast path, a tokenizer core written in Zig and linked from prebuilt static libraries. Dictionaries are compiled once into a binary file that is memory-mapped at load time.
  • Tokenizer: a pure-Rust tokenizer that reads the raw dictionary files directly. Always available, roughly an order of magnitude slower.
  • VibratoSystemDictionary (feature vibrato-system): load precompiled Vibrato system.dic / system.dic.zst files.

See the README for the supported zig-ffi targets and how to prepare IPADIC.

§Quick start

Compile the raw MeCab files (UTF-8 lex.csv, matrix.def, char.def, unk.def) into a binary dictionary once, then memory-map it and tokenize with a reusable worker, reading surfaces and features as borrowed views:

use delarocha::ffi::ZigTokenizer;

fn main() -> delarocha::Result<()> {
    // Once (e.g. at build or deploy time): compile the raw MeCab files.
    ZigTokenizer::write_binary_from_raw_paths(
        "dic/lex.csv",
        "dic/matrix.def",
        "dic/char.def",
        "dic/unk.def",
        "ipadic.dic",
    )?;

    // At startup: memory-map the binary dictionary (nothing is copied).
    let tokenizer = ZigTokenizer::from_binary_path("ipadic.dic")?;

    // Reuse one worker per thread; its buffers are recycled between calls.
    let mut worker = tokenizer.create_worker()?;
    for sentence in ["本とカレーの街", "東京都に住む"] {
        for token in worker.tokenize_borrowed_views(sentence)?.iter() {
            println!("{}\t{}", token.surface(), token.feature());
        }
    }
    Ok(())
}

Without zig-ffi, the pure-Rust tokenizer reads the same raw files:

use std::fs::File;

fn main() -> delarocha::Result<()> {
    let dictionary = delarocha::SystemDictionaryBuilder::from_readers(
        File::open("dic/lex.csv")?,
        File::open("dic/matrix.def")?,
        File::open("dic/char.def")?,
        File::open("dic/unk.def")?,
    )?;
    let tokenizer = delarocha::Tokenizer::new(dictionary);
    let mut worker = tokenizer.create_worker();
    for token in worker.tokenize("本とカレーの街")? {
        println!("{}\t{}", token.surface(), token.feature());
    }
    Ok(())
}

§Binary dictionary compatibility

While delarocha is at 0.x, the binary dictionary format written by ffi::ZigTokenizer::write_binary_from_raw_paths may change between minor versions (0.1 to 0.2, for example); patch releases keep it. The current format is version 4 (magic DLRDIC04). A file written by another format version is rejected at load time with Error::UnsupportedDictionaryVersion and must be rebuilt from the raw MeCab dictionary files, so keep the raw files (or rebuild as part of your build or deployment) rather than shipping only the .dic file.

Modules§

ffizig-ffi

Structs§

CompatWorker
Dictionary
SystemDictionaryBuilder
Token
Tokenizer
VibratoSystemDictionaryvibrato-system
VibratoSystemTokenvibrato-system
VibratoSystemTokenizervibrato-system
VibratoSystemWorkervibrato-system
Worker

Enums§

Error

Type Aliases§

Result