Trait llm_chain::tokens::ExecutorTokenCountExt

pub trait ExecutorTokenCountExt<Output, Token: Clone, StepTokenizer>: Executor<Output = Output, Token = Token> {
    // Provided method
    fn split_to_fit(
        &self,
        step: &Step<Self>,
        doc: &Parameters,
        base_parameters: &Parameters,
        chunk_overlap: Option<usize>
    ) -> Result<Vec<Parameters>, PromptTokensError> { ... }
}

An extension trait for the Executor trait that provides additional methods for working with token counts.

Provided Methods

fn split_to_fit( &self, step: &Step<Self>, doc: &Parameters, base_parameters: &Parameters, chunk_overlap: Option<usize> ) -> Result<Vec<Parameters>, PromptTokensError>

Splits a Parameters object into multiple smaller Parameters objects that fit within the context window size supported by the given model.

Arguments

• step - The step that will process the Parameters. Has impact on tokenizer & text splitter used
• doc - The parameter object to split into multiple, smaller, parameters
• chunk_overlap - The amount of tokens each split part should overlap with previous & next chunk

Errors

Returns a PromptTokensError if there is an issue computing the tokens.

Implementors

impl<E, O, T, N> ExecutorTokenCountExt<O, T, N> for Ewhere E: Executor<Output = O, Token = T>, T: Clone,

An extension trait for the Executor trait that provides additional methods for working with tokens