delarocha
delarocha is a Japanese morphological analyzer (tokenizer) for Rust that
reads MeCab-format system dictionaries such as IPADIC and produces
MeCab/Vibrato-compatible segmentation. Its fast path is a Viterbi tokenizer
core written in Zig, shipped as prebuilt static libraries behind the zig-ffi
feature so you do not need Zig installed; dictionaries are compiled once into a
binary file that loads in milliseconds through a read-only memory map. A
dependency-light pure-Rust tokenizer is always available as a fallback, and
precompiled Vibrato dictionaries can be loaded with the vibrato-system
feature.
Installation
Features:
zig-ffi: link the Zig tokenizer core (delarocha::ffi). Recommended; see Supported targets.vibrato-system: load precompiled Vibratosystem.dic/system.dic.zst.wasm:wasm-bindgenbindings used by the browser playground.
Without features, cargo add delarocha gives the pure-Rust tokenizer.
Quick start
delarocha reads the four raw MeCab dictionary files as UTF-8: lex.csv (all
lexicon CSV files concatenated), matrix.def, char.def, and unk.def. For
mecab-ipadic (EUC-JP), prepare them once:
|
for; do
done
The recommended fast path compiles the raw files into a binary dictionary once, then memory-maps it and tokenizes with a reusable worker, reading surfaces and features as borrowed views (no per-token allocation):
use ZigTokenizer;
While delarocha is 0.x, rebuild ipadic.dic when upgrading to a new minor
version (see Binary dictionary compatibility).
Without zig-ffi, the pure-Rust tokenizer reads the same raw files directly:
use File;
The pure-Rust path is roughly an order of magnitude slower than the Zig core and parses the raw dictionary on every start; use it where the Zig core is not available. To compare throughput with Vibrato on your own machine and text, see Benchmarks.
Supported targets
The zig-ffi feature links a prebuilt static Zig library for common Rust
targets, so downstream crates can use the Rust crate without installing Zig.
The bundled targets are:
aarch64-apple-darwinx86_64-apple-darwinaarch64-unknown-linux-gnux86_64-unknown-linux-gnui686-pc-windows-msvcx86_64-pc-windows-msvcwasm32-unknown-unknown
The WASM artifact is built with Zig's ReleaseSmall optimization mode to keep
the download size low. Native artifacts continue to use ReleaseFast.
Other targets cannot use zig-ffi from the crates.io package, because it does
not contain the Zig sources. Either use the pure-Rust tokenizer, or depend on a
repository checkout (for example a git
dependency) with Zig 0.16 on PATH: unsupported targets then build the static
library from zig/src/lib.zig. Set DELAROCHA_BUILD_ZIG=1 to rebuild it from
source even for bundled targets.
Binary dictionary compatibility
While delarocha is at 0.x, the binary dictionary format written by
ZigTokenizer::write_binary_from_raw_paths may change between minor versions
(0.1 to 0.2, for example); patch releases keep it. The current format is
version 4 (magic DLRDIC04). A file written by another format version is
rejected at load time with Error::UnsupportedDictionaryVersion (Zig:
error.UnsupportedDictionaryVersion) and must be rebuilt from the raw MeCab
dictionary files, so keep the raw files, or rebuild the binary dictionary as
part of your build or deployment, rather than shipping only the .dic file.
Rebuilding the same raw files with the same delarocha version produces
byte-identical output.
Zig tokenizer
The Zig core lives in delarocha::ffi (feature zig-ffi). ZigTokenizer
loads a dictionary (binary, raw MeCab files, or the TSV fixture format) and is
shared read-only; each thread creates its own ZigWorker.
Binary dictionaries loaded from a path (ZigTokenizer::from_binary_path,
ZigTokenizer::count_only_from_binary_path, and Zig's
Tokenizer.initBinaryFile / Dictionary.fromBinaryFile) are memory-mapped
read-only and borrowed for the tokenizer's lifetime: features, the connection
matrix, and all trie tables are used in place instead of being copied. Do not
truncate or rewrite such a file in place while a tokenizer uses it (write a new
file and rename it instead). ZigTokenizer::from_binary_bytes and Zig's
Dictionary.fromBinaryFileCopy keep a private copy when that cannot be
guaranteed.
Binary dictionaries use format version 4 (magic DLRDIC04). Every table
starts at a 16-byte aligned offset and entry features are located through a
separate offset table, so loading a memory-mapped dictionary does not read any
per-entry data: pages are only faulted in when tokenization touches them.
Files written by earlier versions (DLRDIC01, DLRDIC02, DLRDIC03) or by an unknown
format version are rejected with UnsupportedDictionaryVersion (Rust:
Error::UnsupportedDictionaryVersion), and corrupt tables or out-of-range
indices with InvalidDictionary; rebuild them from the raw MeCab dictionary
files with ZigTokenizer::write_binary_from_raw_paths (or Zig's
Dictionary.toBinaryAlloc). Rebuilding the same raw files produces
byte-identical output.
When writing a binary dictionary, the connection (left/right context) ids are
renumbered so that frequently used connection-matrix rows and columns sit next
to each other in memory. By default the order is estimated from the lexicon
alone (ConnectionIdOrder::DictionaryPrior);
ZigTokenizer::write_binary_from_raw_paths_with_id_order can instead keep the
raw ids (Original), rank them by a precomputed id left_weight right_weight
file (WeightsFile), or measure them by tokenizing a sample text
(SampleText; use text representative of, but not identical to, the text you
will tokenize). Renumbering is invisible to callers: token spans, word ids,
features and costs are unchanged, and no API exposes connection ids. On the
Yokohama benchmark it speeds up tokenization by roughly 2-4%.
Large dictionaries store entry features compactly: the leading CSV columns
(for IPADIC the six part-of-speech/conjugation columns) come from a small
shared table, and the remaining columns are encoded as back-references to the
surface, its katakana rendering, or an earlier column plus a literal suffix,
with a raw fallback per entry. The builder picks the layout from the data and
keeps raw features when the encoding would not be smaller (Zig:
Dictionary.toBinaryAllocWithOptions(.{ .compact_features = false }) forces
raw). For IPADIC the feature data shrinks from 31.1 MB to 7.5 MB and the
dictionary file from 63.6 MB to 40.0 MB. Tokens still expose byte-identical
feature strings: when features are requested, each distinct word is decoded
once per tokenizer into a table shared by all of its workers (lock-free
lookups, grown only with the words actually seen, capped at 40 MiB by default,
which holds all of IPADIC; see ZigTokenizer::shared_feature_bytes and
set_shared_feature_limit). Words beyond the cap go to a bounded per-worker
cache. Span/count-only tokenization does not decode anything, and features
borrowed from a worker are valid until the worker is used again.
For output-sensitive callers, ZigWorker::tokenize_borrowed_views returns a
lazy ZigTokenViews collection that reads the worker's native token array in
place: the call itself does no per-token work beyond tokenization (so
views.len() costs the same as tokenize_raw), and features and character
offsets are only computed for the tokens you read. Iterating it avoids both
owned surface/feature strings and the per-call Vec<ZigTokenView> allocation;
iter() is O(1) amortized per token, and get(i) counts characters from the
previously read token, so sequential access is cheap while arbitrary random
access is O(input length) per call in the worst case. The collection and its token views remain valid
until the worker is mutably used again. Each ZigTokenView carries the borrowed
surface and feature plus byte and character ranges (range_byte,
range_char), and to_token() reproduces the owned ZigWorker::tokenize
output exactly.
Callers that need owned tokens can pass a reusable vector to
ZigWorker::tokenize_into; existing Token string buffers are overwritten in
place, so a sentence loop stops allocating once the buffers warm up.
tokenize_spans_into does the same for span-only output.
Feature strings are checked for UTF-8 once per dictionary word id per worker and memoized, rather than on every emitted token.
Worker memory
Workers keep their lattice and token buffers between calls so steady-state tokenization does not allocate. They grow to fit the largest input seen, about 60 bytes per input byte for full tokenization and 32 for count-only on IPADIC (a 659 KB document leaves roughly 39 MB / 20 MB behind). To give that back:
retained_bytes()reports the capacity currently held.shrink_to(max_bytes)frees buffers (largest first) until at mostmax_bytesremain;shrink_to_fit()frees everything. Later calls regrow what they need and return identical tokens.set_retained_capacity_limit(Some(max_bytes))trims automatically after any call that leaves more thanmax_bytesretained. Calls whose buffers stay below the limit never trim, so small-sentence loops keep reusing their buffers. The buffer holding the tokens just returned is kept (shrunk to the result) until a later call replaces it.
These exist on ZigWorker (native lattice, Rust-side token metadata, parked
tokenize_into tokens, and the UTF-8 memo) and on the pure-Rust Worker. In
Zig they are Worker.retainedBytes, Worker.shrinkTo / Worker.shrink, and
Worker.setRetainedCapacityLimit; the C ABI exports
delarocha_worker_retained_bytes, delarocha_worker_shrink_to, and
delarocha_worker_set_retained_limit (SIZE_MAX removes the limit).
Precompiled Vibrato dictionaries
The vibrato-system feature loads precompiled Vibrato dictionaries
directly from system.dic or zstd-compressed system.dic.zst:
let tokenizer = from_path?
.into_tokenizer
.ignore_space?
.max_grouping_len;
let tokens = tokenizer.tokenize?;
Development
The sections below are for working on delarocha itself from a repository checkout.
Implementation overview
- runtime dictionary loader
Tokenizer/ reusableWorker- AoS nodes
- dense row-major connection matrix
- minimal unknown word handling
- C ABI for Rust bindings
cargo benchharness for comparison with Vibrato
Dictionary fixture
The Zig C ABI still accepts a deliberately small TSV fixture format:
matrix <right_size> <left_size>
<row 0 i16 costs...>
entry <surface> <left_id> <right_id> <word_cost> <feature>
The Rust binding now also accepts MeCab/Vibrato-style raw dictionary readers:
lex.csvmatrix.defchar.defunk.def
This matches the input shape of vibrato::SystemDictionaryBuilder::from_readers and is the current compatibility path for differential tests and the CLI.
Rust Tests
The CI workflow runs the Rust, Vibrato system dictionary, and Zig unit tests on
Linux, macOS, and Windows. It also runs zig-ffi tests and compiles the
Yokohama text benchmark on Linux and macOS; Windows currently exercises the
pure Rust, vibrato-system, and Zig test suites while the MSVC Zig FFI link
path is kept out of the matrix.
Fuzzing
The repository has deterministic seeded fuzz-style tests that generate mixed Japanese, ASCII, whitespace, punctuation, and emoji inputs on every CI run:
bindings/rust/tests/fuzz_tokenizer.rsverifies Rust tokenization does not fail and that emitted token spans rebuild the original UTF-8 input.bindings/rust/tests/zig_ffi.rsverifies Zig binary count-only tokenization returns the same token count as full Zig tokenization for seeded random inputs.
Run them locally with:
Stress them with more generated inputs by overriding the deterministic seed count and maximum generated input length:
DELAROCHA_FUZZ_SEEDS=100000 DELAROCHA_FUZZ_MAX_LEN=256
DELAROCHA_FUZZ_SEEDS=100000 DELAROCHA_FUZZ_MAX_LEN=256
Coverage-guided fuzzing uses cargo-fuzz with libFuzzer. The tokenize
target fuzzes valid UTF-8 input against token span and surface invariants, and
the dictionary target fuzzes the compact dictionary parser for panic-free
error handling.
CLI
|
MeCab-compatible space skipping and unknown grouping options are available:
Zig Tests
Requires Zig on PATH.
Zig Core Benchmark
The pure Zig microbenchmark measures tokenizer core time without Rust/C ABI overhead.
Zig FFI tests and prebuilt libraries
To regenerate the checked-in libraries after changing the Zig sources, run
scripts/build_prebuilt.sh (on macOS, so the Darwin archives can be repacked
with 8-byte aligned members); see
bindings/rust/prebuilt/README.md.
Benchmarks
Run the baseline benchmark:
Compare against Vibrato by enabling vibrato-bench and pointing to a compiled Vibrato dictionary. Compressed .zst dictionaries are decompressed through zstd before calling vibrato::Dictionary::read, matching Vibrato's API note.
VIBRATO_SYSTEM_DIC=/path/to/system.dic.zst \
A precompiled ipadic dictionary can be downloaded from Vibrato releases:
VIBRATO_SYSTEM_DIC="/target/vibrato-dic/ipadic-mecab-2_7_0/system.dic.zst" \
The Vibrato project and dictionary release information are available at https://github.com/daac-tools/vibrato.
Vibrato-Style Benchmark
The vibrato_style_benchmark example mirrors the benchmark runner used by
daac-tools/vibrato: it reads newline-separated sentences from stdin, runs
10 warm runs per trial, repeats 10 trials, discards each trial's fastest and
slowest run, and prints Elapsed_seconds_to_tokenize_all_sentences.
Use --full to benchmark full token materialization instead of the count-only
path. The default count-only mode matches Vibrato's benchmark loop shape by
tokenizing each sentence and accumulating token counts without formatting
token output.
--full materializes zero-copy token views (tokenize_borrowed_views: borrowed
surface and feature, byte and character ranges). This is the like-for-like
comparison with Vibrato, whose worker tokens also borrow the sentence and the
dictionary instead of allocating strings. Add --owned (--full --owned) to
measure the owned Vec<Token> API, which allocates two strings per token.
Yokohama Ordinance Text Benchmark
To reproduce the long-text comparison used for the Yokohama City tax ordinance, download the HTML, extract normalized visible text, and run the dedicated example:
YOKOHAMA_TEXT="/target/yokohama/g202RG00000570.txt" \
YOKOHAMA_BENCH_WARMUP=5 \
YOKOHAMA_BENCH_ITERATIONS=50 \
ZIG_RAW_DIC_DIR=/path/to/raw-ipadic \
VIBRATO_SYSTEM_DIC=/path/to/system.dic.zst \
The example prints per-iteration wall-clock time and token count for delarocha/binary-count-only, delarocha/raw-count-only, and vibrato/system-dic on the same extracted text.
Memory Comparison
The Rust example below reports RSS after dictionary load and after repeated tokenization for the fixture dictionary.
For ipadic-scale dictionaries, set the dictionary paths and use the system dictionary modes:
ZIG_RAW_DIC_DIR=/path/to/raw-ipadic \
ZIG_RAW_DIC_DIR=/path/to/raw-ipadic \
VIBRATO_SYSTEM_DIC=/path/to/system.dic.zst \
WebAssembly Playground
The Rust tokenizer can be compiled to WebAssembly and used from a static browser playground. The playground downloads MeCab ipadic source data, converts it to UTF-8, and serves gzip-compressed dictionary assets next to the static UI:
Pushes to main build the same artifact and deploy it to GitHub Pages.
License
Licensed under either of Apache License, Version 2.0 or MIT license at your option.