Crate sentencepiece_model

SentencePiece model parser generated from the SentencePiece protobuf definition.

See SentencePieceModel for the entry point for parsing and accessing sentencepiece models.

Structs

• ModelProto
  ModelProto stores model parameters. SentencePieceProcessor is supposed to be self-contained. All settings/parameters which may change the behavior must be encoded in ModelProto.
• NormalizerSpec
  NormalizerSpec encodes a various parameters for string normalizaiton
• SentencePiece
  SentencePiece piece.
• SentencePieceModel
  SentencePiece model. Provides access to the underlying sentencepiece model.
• TrainerSpec
  TrainerSpec encodes a various parameters for SentencePiece training.

Enums

• ModelType
  Model type. Only UNIGRAM now.
• Type
  Piece type.