Crate vaporetto

Vaporetto

Vaporetto is a fast and lightweight pointwise prediction based tokenizer.

Examples

use std::fs::File;

use vaporetto::{Model, Predictor, Sentence};

let f = File::open("../resources/model.bin")?;
let model = Model::read(f)?;
let predictor = Predictor::new(model, true)?;

let mut buf = String::new();

let mut s = Sentence::default();

s.update_raw("まぁ社長は火星猫だ")?;
predictor.predict(&mut s);
s.fill_tags();
s.write_tokenized_text(&mut buf);
assert_eq!(
    "まぁ/名詞/マー 社長/名詞/シャチョー は/助詞/ワ 火星/名詞/カセー 猫/名詞/ネコ だ/助動詞/ダ",
    buf,
);

s.update_raw("まぁ良いだろう")?;
predictor.predict(&mut s);
s.fill_tags();
s.write_tokenized_text(&mut buf);
assert_eq!(
    "まぁ/副詞/マー 良い/形容詞/ヨイ だろう/助動詞/ダロー",
    buf,
);

Tag prediction requires crate feature tag-prediction.

Training requires crate feature train. For more details, see Trainer.

Modules

• errors
  Definition of errors.

Structs

• KyteaModelkytea
  Model data created by KyTea.
• Model
  Model data.
• Predictor
  Predictor created from the model.
• Sentence
  Sentence data containing boundary and tag annotations.
• Token
  A Token information.
• TokenIterator
  Iterator returned by Sentence::iter_tokens().
• Trainertrain
  Trainer.
• WordWeightRecord
  Record of weights for each word.

Enums

• CharacterBoundary
  Boundary type.
• CharacterType
  Character type.
• SolverTypetrain
  Solver type.

Constants

• VERSION
  Version number of this library.