delarocha 0.1.1

Japanese morphological analyzer (MeCab/Vibrato-compatible tokenizer) in Rust with an optional Zig core
Documentation

delarocha

CI crates.io docs.rs

delarocha is a Japanese morphological analyzer (tokenizer) for Rust that reads MeCab-format system dictionaries such as IPADIC and produces MeCab/Vibrato-compatible segmentation. Its fast path is a Viterbi tokenizer core written in Zig, shipped as prebuilt static libraries behind the zig-ffi feature so you do not need Zig installed; dictionaries are compiled once into a binary file that loads in milliseconds through a read-only memory map. A dependency-light pure-Rust tokenizer is always available as a fallback, and precompiled Vibrato dictionaries can be loaded with the vibrato-system feature.

Installation

cargo add delarocha --features zig-ffi

Features:

  • zig-ffi: link the Zig tokenizer core (delarocha::ffi). Recommended; see Supported targets.
  • vibrato-system: load precompiled Vibrato system.dic / system.dic.zst.
  • wasm: wasm-bindgen bindings used by the browser playground.

Without features, cargo add delarocha gives the pure-Rust tokenizer.

Quick start

delarocha reads the four raw MeCab dictionary files as UTF-8: lex.csv (all lexicon CSV files concatenated), matrix.def, char.def, and unk.def. For mecab-ipadic (EUC-JP), prepare them once:

mkdir -p dic
cat mecab-ipadic-2.7.0-20070801/*.csv | iconv -f EUC-JP -t UTF-8 > dic/lex.csv
for f in matrix.def char.def unk.def; do
  iconv -f EUC-JP -t UTF-8 "mecab-ipadic-2.7.0-20070801/$f" > "dic/$f"
done

The recommended fast path compiles the raw files into a binary dictionary once, then memory-maps it and tokenizes with a reusable worker, reading surfaces and features as borrowed views (no per-token allocation):

use delarocha::ffi::ZigTokenizer;

fn main() -> delarocha::Result<()> {
    // Once (e.g. at build or deploy time): compile the raw MeCab files.
    ZigTokenizer::write_binary_from_raw_paths(
        "dic/lex.csv",
        "dic/matrix.def",
        "dic/char.def",
        "dic/unk.def",
        "ipadic.dic",
    )?;

    // At startup: memory-map the binary dictionary (nothing is copied).
    let tokenizer = ZigTokenizer::from_binary_path("ipadic.dic")?;

    // Reuse one worker per thread; its buffers are recycled between calls.
    let mut worker = tokenizer.create_worker()?;
    for sentence in ["本とカレーの街", "東京都に住む"] {
        for token in worker.tokenize_borrowed_views(sentence)?.iter() {
            println!("{}\t{}", token.surface(), token.feature());
        }
    }
    Ok(())
}

While delarocha is 0.x, rebuild ipadic.dic when upgrading to a new minor version (see Binary dictionary compatibility).

Without zig-ffi, the pure-Rust tokenizer reads the same raw files directly:

use std::fs::File;

fn main() -> delarocha::Result<()> {
    let dictionary = delarocha::SystemDictionaryBuilder::from_readers(
        File::open("dic/lex.csv")?,
        File::open("dic/matrix.def")?,
        File::open("dic/char.def")?,
        File::open("dic/unk.def")?,
    )?;
    let tokenizer = delarocha::Tokenizer::new(dictionary);
    let mut worker = tokenizer.create_worker();
    for token in worker.tokenize("本とカレーの街")? {
        println!("{}\t{}", token.surface(), token.feature());
    }
    Ok(())
}

The pure-Rust path is roughly an order of magnitude slower than the Zig core and parses the raw dictionary on every start; use it where the Zig core is not available. To compare throughput with Vibrato on your own machine and text, see Benchmarks.

Supported targets

The zig-ffi feature links a prebuilt static Zig library for common Rust targets, so downstream crates can use the Rust crate without installing Zig. The bundled targets are:

  • aarch64-apple-darwin
  • x86_64-apple-darwin
  • aarch64-unknown-linux-gnu
  • x86_64-unknown-linux-gnu
  • i686-pc-windows-msvc
  • x86_64-pc-windows-msvc
  • wasm32-unknown-unknown

The WASM artifact is built with Zig's ReleaseSmall optimization mode to keep the download size low. Native artifacts continue to use ReleaseFast.

Other targets cannot use zig-ffi from the crates.io package, because it does not contain the Zig sources. Either use the pure-Rust tokenizer, or depend on a repository checkout (for example a git dependency) with Zig 0.16 on PATH: unsupported targets then build the static library from zig/src/lib.zig. Set DELAROCHA_BUILD_ZIG=1 to rebuild it from source even for bundled targets.

Binary dictionary compatibility

While delarocha is at 0.x, the binary dictionary format written by ZigTokenizer::write_binary_from_raw_paths may change between minor versions (0.1 to 0.2, for example); patch releases keep it. The current format is version 4 (magic DLRDIC04). A file written by another format version is rejected at load time with Error::UnsupportedDictionaryVersion (Zig: error.UnsupportedDictionaryVersion) and must be rebuilt from the raw MeCab dictionary files, so keep the raw files, or rebuild the binary dictionary as part of your build or deployment, rather than shipping only the .dic file. Rebuilding the same raw files with the same delarocha version produces byte-identical output.

Zig tokenizer

The Zig core lives in delarocha::ffi (feature zig-ffi). ZigTokenizer loads a dictionary (binary, raw MeCab files, or the TSV fixture format) and is shared read-only; each thread creates its own ZigWorker.

Binary dictionaries loaded from a path (ZigTokenizer::from_binary_path, ZigTokenizer::count_only_from_binary_path, and Zig's Tokenizer.initBinaryFile / Dictionary.fromBinaryFile) are memory-mapped read-only and borrowed for the tokenizer's lifetime: features, the connection matrix, and all trie tables are used in place instead of being copied. Do not truncate or rewrite such a file in place while a tokenizer uses it (write a new file and rename it instead). ZigTokenizer::from_binary_bytes and Zig's Dictionary.fromBinaryFileCopy keep a private copy when that cannot be guaranteed.

Binary dictionaries use format version 4 (magic DLRDIC04). Every table starts at a 16-byte aligned offset and entry features are located through a separate offset table, so loading a memory-mapped dictionary does not read any per-entry data: pages are only faulted in when tokenization touches them. Files written by earlier versions (DLRDIC01, DLRDIC02, DLRDIC03) or by an unknown format version are rejected with UnsupportedDictionaryVersion (Rust: Error::UnsupportedDictionaryVersion), and corrupt tables or out-of-range indices with InvalidDictionary; rebuild them from the raw MeCab dictionary files with ZigTokenizer::write_binary_from_raw_paths (or Zig's Dictionary.toBinaryAlloc). Rebuilding the same raw files produces byte-identical output.

When writing a binary dictionary, the connection (left/right context) ids are renumbered so that frequently used connection-matrix rows and columns sit next to each other in memory. By default the order is estimated from the lexicon alone (ConnectionIdOrder::DictionaryPrior); ZigTokenizer::write_binary_from_raw_paths_with_id_order can instead keep the raw ids (Original), rank them by a precomputed id left_weight right_weight file (WeightsFile), or measure them by tokenizing a sample text (SampleText; use text representative of, but not identical to, the text you will tokenize). Renumbering is invisible to callers: token spans, word ids, features and costs are unchanged, and no API exposes connection ids. On the Yokohama benchmark it speeds up tokenization by roughly 2-4%.

Large dictionaries store entry features compactly: the leading CSV columns (for IPADIC the six part-of-speech/conjugation columns) come from a small shared table, and the remaining columns are encoded as back-references to the surface, its katakana rendering, or an earlier column plus a literal suffix, with a raw fallback per entry. The builder picks the layout from the data and keeps raw features when the encoding would not be smaller (Zig: Dictionary.toBinaryAllocWithOptions(.{ .compact_features = false }) forces raw). For IPADIC the feature data shrinks from 31.1 MB to 7.5 MB and the dictionary file from 63.6 MB to 40.0 MB. Tokens still expose byte-identical feature strings: when features are requested, each distinct word is decoded once per tokenizer into a table shared by all of its workers (lock-free lookups, grown only with the words actually seen, capped at 40 MiB by default, which holds all of IPADIC; see ZigTokenizer::shared_feature_bytes and set_shared_feature_limit). Words beyond the cap go to a bounded per-worker cache. Span/count-only tokenization does not decode anything, and features borrowed from a worker are valid until the worker is used again.

For output-sensitive callers, ZigWorker::tokenize_borrowed_views returns a lazy ZigTokenViews collection that reads the worker's native token array in place: the call itself does no per-token work beyond tokenization (so views.len() costs the same as tokenize_raw), and features and character offsets are only computed for the tokens you read. Iterating it avoids both owned surface/feature strings and the per-call Vec<ZigTokenView> allocation; iter() is O(1) amortized per token, and get(i) counts characters from the previously read token, so sequential access is cheap while arbitrary random access is O(input length) per call in the worst case. The collection and its token views remain valid until the worker is mutably used again. Each ZigTokenView carries the borrowed surface and feature plus byte and character ranges (range_byte, range_char), and to_token() reproduces the owned ZigWorker::tokenize output exactly.

Callers that need owned tokens can pass a reusable vector to ZigWorker::tokenize_into; existing Token string buffers are overwritten in place, so a sentence loop stops allocating once the buffers warm up. tokenize_spans_into does the same for span-only output.

Feature strings are checked for UTF-8 once per dictionary word id per worker and memoized, rather than on every emitted token.

Worker memory

Workers keep their lattice and token buffers between calls so steady-state tokenization does not allocate. They grow to fit the largest input seen, about 60 bytes per input byte for full tokenization and 32 for count-only on IPADIC (a 659 KB document leaves roughly 39 MB / 20 MB behind). To give that back:

  • retained_bytes() reports the capacity currently held.
  • shrink_to(max_bytes) frees buffers (largest first) until at most max_bytes remain; shrink_to_fit() frees everything. Later calls regrow what they need and return identical tokens.
  • set_retained_capacity_limit(Some(max_bytes)) trims automatically after any call that leaves more than max_bytes retained. Calls whose buffers stay below the limit never trim, so small-sentence loops keep reusing their buffers. The buffer holding the tokens just returned is kept (shrunk to the result) until a later call replaces it.

These exist on ZigWorker (native lattice, Rust-side token metadata, parked tokenize_into tokens, and the UTF-8 memo) and on the pure-Rust Worker. In Zig they are Worker.retainedBytes, Worker.shrinkTo / Worker.shrink, and Worker.setRetainedCapacityLimit; the C ABI exports delarocha_worker_retained_bytes, delarocha_worker_shrink_to, and delarocha_worker_set_retained_limit (SIZE_MAX removes the limit).

Precompiled Vibrato dictionaries

The vibrato-system feature loads precompiled Vibrato dictionaries directly from system.dic or zstd-compressed system.dic.zst:

let tokenizer = delarocha::VibratoSystemDictionary::from_path("system.dic.zst")?
    .into_tokenizer()
    .ignore_space(true)?
    .max_grouping_len(24);
let tokens = tokenizer.tokenize("これはテストです。")?;

Development

The sections below are for working on delarocha itself from a repository checkout.

Implementation overview

  • runtime dictionary loader
  • Tokenizer / reusable Worker
  • AoS nodes
  • dense row-major connection matrix
  • minimal unknown word handling
  • C ABI for Rust bindings
  • cargo bench harness for comparison with Vibrato

Dictionary fixture

The Zig C ABI still accepts a deliberately small TSV fixture format:

matrix	<right_size>	<left_size>
<row 0 i16 costs...>
entry	<surface>	<left_id>	<right_id>	<word_cost>	<feature>

The Rust binding now also accepts MeCab/Vibrato-style raw dictionary readers:

  • lex.csv
  • matrix.def
  • char.def
  • unk.def

This matches the input shape of vibrato::SystemDictionaryBuilder::from_readers and is the current compatibility path for differential tests and the CLI.

Rust Tests

cargo test

The CI workflow runs the Rust, Vibrato system dictionary, and Zig unit tests on Linux, macOS, and Windows. It also runs zig-ffi tests and compiles the Yokohama text benchmark on Linux and macOS; Windows currently exercises the pure Rust, vibrato-system, and Zig test suites while the MSVC Zig FFI link path is kept out of the matrix.

Fuzzing

The repository has deterministic seeded fuzz-style tests that generate mixed Japanese, ASCII, whitespace, punctuation, and emoji inputs on every CI run:

  • bindings/rust/tests/fuzz_tokenizer.rs verifies Rust tokenization does not fail and that emitted token spans rebuild the original UTF-8 input.
  • bindings/rust/tests/zig_ffi.rs verifies Zig binary count-only tokenization returns the same token count as full Zig tokenization for seeded random inputs.

Run them locally with:

cargo test -p delarocha
cargo test -p delarocha --features zig-ffi

Stress them with more generated inputs by overriding the deterministic seed count and maximum generated input length:

DELAROCHA_FUZZ_SEEDS=100000 DELAROCHA_FUZZ_MAX_LEN=256 cargo test -p delarocha --test fuzz_tokenizer
DELAROCHA_FUZZ_SEEDS=100000 DELAROCHA_FUZZ_MAX_LEN=256 cargo test -p delarocha --features zig-ffi --test zig_ffi

Coverage-guided fuzzing uses cargo-fuzz with libFuzzer. The tokenize target fuzzes valid UTF-8 input against token span and surface invariants, and the dictionary target fuzzes the compact dictionary parser for panic-free error handling.

cargo install cargo-fuzz
cargo +nightly fuzz run tokenize -- -max_total_time=30
cargo +nightly fuzz run dictionary -- -max_total_time=30

CLI

echo '本とカレー' | cargo run -p delarocha -- \
  --lex fixtures/lex.csv \
  --matrix fixtures/matrix.def \
  --char fixtures/char.def \
  --unk fixtures/unk.def \
  -O wakati

MeCab-compatible space skipping and unknown grouping options are available:

cargo run -p delarocha -- --lex lex.csv --matrix matrix.def --char char.def --unk unk.def -S -M 24

Zig Tests

Requires Zig on PATH.

cd zig
zig build test

Zig Core Benchmark

The pure Zig microbenchmark measures tokenizer core time without Rust/C ABI overhead.

cd zig
zig build bench -Doptimize=ReleaseFast

Zig FFI tests and prebuilt libraries

cargo test -p delarocha --features zig-ffi

To regenerate the checked-in libraries after changing the Zig sources, run scripts/build_prebuilt.sh (on macOS, so the Darwin archives can be repacked with 8-byte aligned members); see bindings/rust/prebuilt/README.md.

Benchmarks

Run the baseline benchmark:

cargo bench -p delarocha

Compare against Vibrato by enabling vibrato-bench and pointing to a compiled Vibrato dictionary. Compressed .zst dictionaries are decompressed through zstd before calling vibrato::Dictionary::read, matching Vibrato's API note.

VIBRATO_SYSTEM_DIC=/path/to/system.dic.zst \
  cargo bench -p delarocha --features vibrato-bench

A precompiled ipadic dictionary can be downloaded from Vibrato releases:

mkdir -p target/vibrato-dic
curl -L -o target/vibrato-dic/ipadic-mecab-2_7_0.tar.xz \
  https://github.com/daac-tools/vibrato/releases/download/v0.5.0/ipadic-mecab-2_7_0.tar.xz
tar -xf target/vibrato-dic/ipadic-mecab-2_7_0.tar.xz -C target/vibrato-dic
VIBRATO_SYSTEM_DIC="$PWD/target/vibrato-dic/ipadic-mecab-2_7_0/system.dic.zst" \
  cargo bench -p delarocha --features 'zig-ffi vibrato-bench' --bench tokenize

The Vibrato project and dictionary release information are available at https://github.com/daac-tools/vibrato.

Vibrato-Style Benchmark

The vibrato_style_benchmark example mirrors the benchmark runner used by daac-tools/vibrato: it reads newline-separated sentences from stdin, runs 10 warm runs per trial, repeats 10 trials, discards each trial's fastest and slowest run, and prints Elapsed_seconds_to_tokenize_all_sentences.

cargo run -p delarocha --release --features zig-ffi \
  --example vibrato_style_benchmark -- \
  -i /path/to/delarocha.dic < test.txt

Use --full to benchmark full token materialization instead of the count-only path. The default count-only mode matches Vibrato's benchmark loop shape by tokenizing each sentence and accumulating token counts without formatting token output.

--full materializes zero-copy token views (tokenize_borrowed_views: borrowed surface and feature, byte and character ranges). This is the like-for-like comparison with Vibrato, whose worker tokens also borrow the sentence and the dictionary instead of allocating strings. Add --owned (--full --owned) to measure the owned Vec<Token> API, which allocates two strings per token.

Yokohama Ordinance Text Benchmark

To reproduce the long-text comparison used for the Yokohama City tax ordinance, download the HTML, extract normalized visible text, and run the dedicated example:

mkdir -p target/yokohama
curl -L \
  https://cgi.city.yokohama.lg.jp/somu/reiki/reiki_honbun/g202RG00000570.html \
  -o target/yokohama/g202RG00000570.html
python3 scripts/extract_yokohama_reiki_text.py \
  target/yokohama/g202RG00000570.html \
  target/yokohama/g202RG00000570.txt
YOKOHAMA_TEXT="$PWD/target/yokohama/g202RG00000570.txt" \
YOKOHAMA_BENCH_WARMUP=5 \
YOKOHAMA_BENCH_ITERATIONS=50 \
ZIG_RAW_DIC_DIR=/path/to/raw-ipadic \
VIBRATO_SYSTEM_DIC=/path/to/system.dic.zst \
  cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example yokohama_text_bench

The example prints per-iteration wall-clock time and token count for delarocha/binary-count-only, delarocha/raw-count-only, and vibrato/system-dic on the same extracted text.

Memory Comparison

The Rust example below reports RSS after dictionary load and after repeated tokenization for the fixture dictionary.

cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example memory -- delarocha-zig
cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example memory -- vibrato
cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example memory -- delarocha-rust

For ipadic-scale dictionaries, set the dictionary paths and use the system dictionary modes:

ZIG_RAW_DIC_DIR=/path/to/raw-ipadic \
  cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example memory -- delarocha-zig-ipadic-raw
ZIG_RAW_DIC_DIR=/path/to/raw-ipadic \
  cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example memory -- delarocha-zig-ipadic-binary
VIBRATO_SYSTEM_DIC=/path/to/system.dic.zst \
  cargo run -p delarocha --release --features 'zig-ffi vibrato-bench' --example memory -- vibrato-system

WebAssembly Playground

The Rust tokenizer can be compiled to WebAssembly and used from a static browser playground. The playground downloads MeCab ipadic source data, converts it to UTF-8, and serves gzip-compressed dictionary assets next to the static UI:

rustup target add wasm32-unknown-unknown
cargo install -f wasm-bindgen-cli --version 0.2.120
cargo build -p delarocha --lib --target wasm32-unknown-unknown --features wasm --release
mkdir -p public/pkg
cp -R playground/. public/
python3 scripts/prepare_ipadic_playground.py --out-dir public/dic
wasm-bindgen \
  --target web \
  --out-dir public/pkg \
  --out-name delarocha \
  target/wasm32-unknown-unknown/release/delarocha.wasm
python3 -m http.server 4173 --directory public

Pushes to main build the same artifact and deploy it to GitHub Pages.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.