sentencepiece-model 0.1.4

SentencePiece model parser generated from the SentencePiece protobuf definition

Coverage
88.04%
81 out of 92 items documented1 out of 11 items with examples
Size
Source code size: 40.39 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 5.92 MB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 24s Average build duration of successful builds.
all releases: 24s Average build duration of successful builds in releases after 2024-10-23.
Links
Systemcluster/sentencepiece-model
0 0 0
crates.io
Dependencies
Versions
Owners

sentencepiece-model

SentencePiece model parser generated from the SentencePiece protobuf definition.

use sentencepiece_model::SentencePieceModel;

let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer()?.unk_id(), 2);

Usage

[dependencies]
sentencepiece-model = "0.1"

sentencepiece-model uses prost-build and protox to generate Rust code from the SentencePiece protobuf definition at build time. protoc is not required.