Skip to main content

Crate litsea_binding_core

Crate litsea_binding_core 

Source
Expand description

FFI-independent helpers shared by the Litsea language bindings (litsea-python, litsea-nodejs, litsea-php, litsea-ruby, litsea-wasm).

Each binding wraps the litsea crate for its own FFI layer (PyO3, napi, ext-php-rs, magnus, wasm-bindgen). Without a shared layer, all five would reimplement the same plumbing: parsing a language name, deciding whether a model file is a segmentation model or a two-stage POS model, keeping a reusable litsea::SegmentBuffer, attaching byte offsets to tokens, driving the trainers with a cancellation flag, and turning litsea::LitseaError into something the host language can raise. That logic lives here as plain Rust, so it can be unit-tested without any FFI toolchain.

§Model loading

Models are never embedded in a binding package (they range from 86 KB to 8.3 MB); the caller supplies one as bytes, a filesystem path, or a URI:

use litsea::Language;
use litsea_binding_core::CoreSegmenter;

let segmenter = CoreSegmenter::from_path(Language::Japanese, "japanese.model".as_ref())?;
assert!(!segmenter.has_pos());
println!("{:?}", segmenter.segment("すもももももももものうち"));

The model kind is detected from the file itself (litsea::ModelKind::detect), so bindings do not expose a --pos-style flag: loading japanese_pos.model yields a segmenter where CoreSegmenter::has_pos is true and CoreSegmenter::segment_with_pos works.

§Platform support

trainer and the blocking model loaders are unavailable on wasm32-unknown-unknown: feature extraction and training are file-based, and there is no filesystem. WASM callers fetch the model bytes in JavaScript and use CoreSegmenter::from_bytes.

Re-exports§

pub use cancel::CancelToken;
pub use error::CoreError;
pub use error::CoreResult;
pub use error::ErrorKind;
pub use language::SUPPORTED_LANGUAGES;
pub use language::language_name;
pub use language::parse_language;
pub use language::supported_language_names;
pub use model::read_model_file;
pub use model::BuiltSegmenter;
pub use model::build_segmenter;
pub use model::build_segmenter_with;
pub use model::build_user_dictionary;
pub use model::build_user_dictionary_with_tags;
pub use model::read_model_uri;
pub use segmenter::CoreSegmenter;
pub use token::TokenView;
pub use trainer::CoreExtractor;
pub use trainer::CorePerceptronTrainer;
pub use trainer::CoreTrainer;
pub use trainer::CoreTwoStageTrainer;
pub use trainer::CorpusFormat;
pub use trainer::parse_feature_set;

Modules§

cancel
Cooperative cancellation for long-running training runs.
error
Shared, FFI-independent error type for the language bindings.
language
Language-name handling shared by the bindings.
model
Model loading and model-kind dispatch.
runtime
Blocking wrappers for litsea’s async model loading.
segmenter
The segmenter facade the bindings wrap.
token
FFI-independent token representation.
trainer
Feature extraction and model training for the bindings.

Functions§

version
Returns the version of the litsea-binding-core crate.