Crate litsea 

Litsea is an extremely compact word segmentation and POS tagging library implemented in Rust.

It performs word segmentation using a compact pre-trained model in the AdaBoost text format, inspired by TinySegmenter and TinySegmenterMaker; the bundled segmentation models are trained via a lossless Averaged Perceptron collapse rather than AdaBoost boosting (see the litsea::trainer module docs), but the format and inference path are unchanged. It also supports word segmentation and POS (Part-of-Speech) tagging with Universal POS (UPOS) tags through a two-stage architecture: a binary boundary classifier plus a word-level tagger (see the two_stage module).

Supported Languages

• Japanese
• Chinese (Simplified and Traditional)
• Korean
• English

Re-exports

pub use adaboost::AdaBoost;

pub use error::LitseaError;

pub use error::Result;

pub use evaluation::PosMetrics;

pub use evaluation::SegmentationMetrics;

pub use extractor::Extractor;

pub use language::Language;

pub use language::ParseLanguageError;

pub use metrics::BinaryMetrics;

pub use metrics::MulticlassMetrics;

pub use perceptron::AveragedPerceptron;

pub use segmenter::SegmentBuffer;

pub use segmenter::Segmenter;

pub use trainer::PerceptronTrainer;

pub use trainer::Trainer;

pub use trainer::TwoStageMetrics;

pub use trainer::TwoStageTrainer;

pub use two_stage::ModelKind;

pub use two_stage::ParseTwoStageFeatureSetError;

pub use two_stage::TwoStageFeatureSet;

pub use two_stage::TwoStageLearner;

pub use upos::ParseSegmentLabelError;

pub use upos::ParseUposError;

pub use upos::SegmentLabel;

pub use upos::Upos;

Modules

adaboost

AdaBoost binary classifier for word-boundary prediction.

error

Error types for the litsea library.

evaluation

Held-out evaluation of segmentation (and POS tagging) quality.

extractor

Feature extraction from training corpora.

language

Supported languages and character type classification.

metrics

Evaluation metrics for the learners.

model_io

Shared model loading I/O.

perceptron

Multiclass Averaged Perceptron, the training-side learner behind both stages of the two-stage architecture (issue #147) and the bundled segmentation models’ collapse recipe.

segmenter

Text segmentation engine.

trainer

High-level training front-ends.

two_stage

Two-stage model container and file format (litsea-two-stage v1).

upos

Universal POS tags and segmentation labels.

Functions

version

Returns the version of the litsea crate (the CARGO_PKG_VERSION it was built with), e.g. "0.6.0".