Expand description
FFI-independent helpers shared by the Litsea language bindings
(litsea-python, litsea-nodejs, litsea-php, litsea-ruby,
litsea-wasm).
Each binding wraps the litsea crate for its own FFI layer (PyO3,
napi, ext-php-rs, magnus, wasm-bindgen). Without a shared layer, all five
would reimplement the same plumbing: parsing a language name, deciding
whether a model file is a segmentation model or a two-stage POS model,
keeping a reusable litsea::SegmentBuffer, attaching byte offsets to
tokens, driving the trainers with a cancellation flag, and turning
litsea::LitseaError into something the host language can raise. That
logic lives here as plain Rust, so it can be unit-tested without any FFI
toolchain.
§Model loading
Models are never embedded in a binding package (they range from 86 KB to 8.3 MB); the caller supplies one as bytes, a filesystem path, or a URI:
use litsea::Language;
use litsea_binding_core::CoreSegmenter;
let segmenter = CoreSegmenter::from_path(Language::Japanese, "japanese.model".as_ref())?;
assert!(!segmenter.has_pos());
println!("{:?}", segmenter.segment("すもももももももものうち"));The model kind is detected from the file itself
(litsea::ModelKind::detect), so bindings do not expose a --pos-style
flag: loading japanese_pos.model yields a segmenter where
CoreSegmenter::has_pos is true and
CoreSegmenter::segment_with_pos works.
§Platform support
trainer and the blocking model loaders are unavailable on
wasm32-unknown-unknown: feature extraction and training are file-based,
and there is no filesystem. WASM callers fetch the model bytes in
JavaScript and use CoreSegmenter::from_bytes.
Re-exports§
pub use cancel::CancelToken;pub use error::CoreError;pub use error::CoreResult;pub use error::ErrorKind;pub use language::SUPPORTED_LANGUAGES;pub use language::language_name;pub use language::parse_language;pub use language::supported_language_names;pub use model::read_model_file;pub use model::BuiltSegmenter;pub use model::build_segmenter;pub use model::build_segmenter_with;pub use model::build_user_dictionary;pub use model::read_model_uri;pub use segmenter::CoreSegmenter;pub use token::TokenView;pub use trainer::CoreExtractor;pub use trainer::CorePerceptronTrainer;pub use trainer::CoreTrainer;pub use trainer::CoreTwoStageTrainer;pub use trainer::CorpusFormat;pub use trainer::parse_feature_set;
Modules§
- cancel
- Cooperative cancellation for long-running training runs.
- error
- Shared, FFI-independent error type for the language bindings.
- language
- Language-name handling shared by the bindings.
- model
- Model loading and model-kind dispatch.
- runtime
- Blocking wrappers for
litsea’s async model loading. - segmenter
- The segmenter facade the bindings wrap.
- token
- FFI-independent token representation.
- trainer
- Feature extraction and model training for the bindings.
Functions§
- version
- Returns the version of the
litsea-binding-corecrate.