Crate rten_text

This crate provides tools for pre and post-processing text inputs and outputs of models. This primarily means tokenizing and de-tokenizing text.

If you need a more featureful set of tokenizers, see the tokenizers project.

Modules

• normalizer
• tokenizers
  Module containing tokenizers for converting text into sequences of token IDs that can be fed into models.