Expand description
Batch encoding, special-token insertion, padding, and truncation.
These conveniences layer on top of the raw Tokenizer::encode output.
They are scheme-agnostic (they work for both BPE and WordPiece) and live in
the shared Tokenizer trait as provided methods, so every tokenizer gets
them for free.
Structs§
- Encode
Config - Configuration for
encode_with/encode_batch. - Encoding
- The result of encoding a single sequence with an
EncodeConfig.
Enums§
- Padding
- How a sequence (or batch of sequences) should be padded to a common length.
- Truncation
- How an over-long sequence should be shortened.
Traits§
- Tokenizer
Ext - Higher-level encoding conveniences (special tokens, padding, truncation,
batching) available on every
Tokenizer.