Expand description
Japanese morphological analyzer that reads MeCab-format system dictionaries (such as IPADIC) and produces MeCab/Vibrato-compatible segmentation.
ffi(featurezig-ffi): the fast path, a tokenizer core written in Zig and linked from prebuilt static libraries. Dictionaries are compiled once into a binary file that is memory-mapped at load time.Tokenizer: a pure-Rust tokenizer that reads the raw dictionary files directly. Always available, roughly an order of magnitude slower.VibratoSystemDictionary(featurevibrato-system): load precompiled Vibratosystem.dic/system.dic.zstfiles.
See the README for the
supported zig-ffi targets and how to prepare IPADIC.
§Quick start
Compile the raw MeCab files (UTF-8 lex.csv, matrix.def, char.def,
unk.def) into a binary dictionary once, then memory-map it and tokenize
with a reusable worker, reading surfaces and features as borrowed views:
use delarocha::ffi::ZigTokenizer;
fn main() -> delarocha::Result<()> {
// Once (e.g. at build or deploy time): compile the raw MeCab files.
ZigTokenizer::write_binary_from_raw_paths(
"dic/lex.csv",
"dic/matrix.def",
"dic/char.def",
"dic/unk.def",
"ipadic.dic",
)?;
// At startup: memory-map the binary dictionary (nothing is copied).
let tokenizer = ZigTokenizer::from_binary_path("ipadic.dic")?;
// Reuse one worker per thread; its buffers are recycled between calls.
let mut worker = tokenizer.create_worker()?;
for sentence in ["本とカレーの街", "東京都に住む"] {
for token in worker.tokenize_borrowed_views(sentence)?.iter() {
println!("{}\t{}", token.surface(), token.feature());
}
}
Ok(())
}Without zig-ffi, the pure-Rust tokenizer reads the same raw files:
use std::fs::File;
fn main() -> delarocha::Result<()> {
let dictionary = delarocha::SystemDictionaryBuilder::from_readers(
File::open("dic/lex.csv")?,
File::open("dic/matrix.def")?,
File::open("dic/char.def")?,
File::open("dic/unk.def")?,
)?;
let tokenizer = delarocha::Tokenizer::new(dictionary);
let mut worker = tokenizer.create_worker();
for token in worker.tokenize("本とカレーの街")? {
println!("{}\t{}", token.surface(), token.feature());
}
Ok(())
}§Binary dictionary compatibility
While delarocha is at 0.x, the binary dictionary format written by
ffi::ZigTokenizer::write_binary_from_raw_paths may change between minor
versions (0.1 to 0.2, for example); patch releases keep it. The current
format is version 4 (magic DLRDIC04). A file written by another format
version is rejected at load time with
Error::UnsupportedDictionaryVersion and must be rebuilt from the raw
MeCab dictionary files, so keep the raw files (or rebuild as part of your
build or deployment) rather than shipping only the .dic file.
Modules§
- ffi
zig-ffi
Structs§
- Compat
Worker - Dictionary
- System
Dictionary Builder - Token
- Tokenizer
- Vibrato
System Dictionary vibrato-system - Vibrato
System Token vibrato-system - Vibrato
System Tokenizer vibrato-system - Vibrato
System Worker vibrato-system - Worker