rust-tinysegmenter
Rust implementation of TinySegmenter library, which is a compact Japanese tokenizer.
Install
Adding the following to the Cargo.toml in your project:
[]
= "*"
and import using extern crate
:
extern crate tinysegmenter;
Usage
let tokens = tokenize;
println!; // 私|の|名前|は|中野|です
License
Copyright (c) 2015 woxtu
Licensed under the MIT license.