rust-tinysegmenter
Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.
Install
Adding the following to the Cargo.toml in your project:
[]
= "0.1"
and import using extern crate
:
extern crate tinysegmenter;
Usage
let tokens = tokenize;
println!; // 私|の|名前|は|中野|です
License
Copyright (c) 2015 woxtu
Licensed under the MIT license.