Crate rten_generate 

Utilities to simplify running auto-regressive RTen models such as transformer decoders.

For working examples, see the examples in the rten-examples crate which import rten_generate.

Re-exports

pub use generator::Generator;

pub use generator::GeneratorConfig;

pub use generator::GeneratorError;

pub use generator::GeneratorUtils;

pub use generator::ModelInputsConfig;

Modules

filter

Filters for processing model outputs prior to sampling.

generator

Tools to run the generation loop for an auto-regressive model.

metrics

Record timing metrics during generation.

model

Abstraction over rten::Model for querying and executing ML models.

sampler

Samplers which select a token from model outputs.

text_decoder

Iterator adapters to decode token IDs into text using rten-text.