Crate sentencepiece_model

Source
Expand description

SentencePiece model parser generated from the SentencePiece protobuf definition.

See SentencePieceModel for the entry point for parsing and accessing sentencepiece models.

use sentencepiece_model::SentencePieceModel;

let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer().unwrap().unk_id(), 2);

Structs§

ModelProto
ModelProto stores model parameters. SentencePieceProcessor is supposed to be self-contained. All settings/parameters which may change the behavior must be encoded in ModelProto.
NormalizerSpec
NormalizerSpec encodes a various parameters for string normalizaiton
Sample
Self-test data sample.
SelfTestData
Proto to store samples for self-testing.
SentencePiece
SentencePiece piece.
SentencePieceModel
SentencePiece model. Provides access to the underlying sentencepiece model.
TrainerSpec
TrainerSpec encodes a various parameters for SentencePiece training.

Enums§

ModelType
Model type.
Type
Piece type.