tinysegmenter 0.1.1

Compact Japanese tokenizer
Documentation
  • Coverage
  • 0%
    0 out of 2 items documented0 out of 1 items with examples
  • Size
  • Source code size: 37.09 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 1.12 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 11s Average build duration of successful builds.
  • all releases: 11s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • woxtu/rust-tinysegmenter
    15 3 3
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • woxtu

rust-tinysegmenter

Crates.io

Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.

Install

Adding the following to the Cargo.toml in your project:

[dependencies]
tinysegmenter = "0.1"

and import using extern crate:

extern crate tinysegmenter;

Usage

let tokens = tinysegmenter::tokenize("私の名前は中野です");
println!("{}", &tokens.join("|")); // 私|の|名前|は|中野|です

License

Copyright (c) 2015 woxtu

Licensed under the MIT license.