Module rust_tokenizers::adapters

Adapter helpers to load datasets

While this crate does not aim at providing built-in support for loading dataset, it exposes a few adapters for testing and benchmarking purposes (e.g. for SST2 sentence classification)

Structs

Example

SST2 sample

Enums

Label

Sentiment analysis label

Functions

read_sst2

Reads a SST2 dataset file and returns a vector of SST2 examples