vibrato
Vibrato is a fast implementation of tokenization (or morphological analysis) based on the viterbi algorithm.
Examples
use File;
use ;
use ;
let file = open.unwrap;
let dict = read.unwrap;
let tokenizer = new;
let mut worker = tokenizer.new_worker;
worker.reset_sentence.unwrap;
worker.tokenize;
assert_eq!;
let t0 = worker.token;
assert_eq!;
assert_eq!;
assert_eq!;
assert_eq!;
let t1 = worker.token;
assert_eq!;
assert_eq!;
assert_eq!;
assert_eq!;
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.