Module rust_tokenizers::adapters

Adapter helpers to load datasets

While this crate does not aim at providing built-in support for loading dataset, it exposes a few adapters for testing and benchmarking purposes (e.g. for SST2 sentence classification)

Structs

• Example
  SST2 sample

Enums

• Label
  Sentiment analysis label

Functions

• read_sst2
  Reads a SST2 dataset file and returns a vector of SST2 examples