batch_mode

Trait LanguageModelBatchWorkflow

Source 
pub trait LanguageModelBatchWorkflow<E>:
    FinishProcessingUncompletedBatches<Error = E>
    + ComputeLanguageModelRequests
    + ProcessBatchRequests<Error = E>
where
    E: From<LanguageModelBatchCreationError>,
{
    const REQUESTS_PER_BATCH: usize = 80usize;

    // Required method
    fn plant_seed_and_wait<'life0, 'life1, 'async_trait>(
        &'life0 mut self,
        input_tokens: &'life1 [Self::Seed],
    ) -> Pin<Box<dyn Future<Output = Result<(), E>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             'life1: 'async_trait,
             Self: 'async_trait;

    // Provided method
    fn execute_language_model_batch_workflow<'life0, 'life1, 'async_trait>(
        &'life0 mut self,
        model: LanguageModelType,
        expected_content_type: ExpectedContentType,
        input_tokens: &'life1 [Self::Seed],
    ) -> Pin<Box<dyn Future<Output = Result<(), E>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             'life1: 'async_trait,
             Self: Send + 'async_trait { ... }
}
Expand description

Trait describing a more general “batch workflow” specialized to GPT expansions.

This approach can unify:

  • Reconciling partial/incomplete state,
  • Computing new requests,
  • Chunking them,
  • Sending them to a remote server,
  • Handling the results.

Provided Associated Constants§

Source

const REQUESTS_PER_BATCH: usize = 80usize

Required Methods§

Source

fn plant_seed_and_wait<'life0, 'life1, 'async_trait>( &'life0 mut self, input_tokens: &'life1 [Self::Seed], ) -> Pin<Box<dyn Future<Output = Result<(), E>> + Send + 'async_trait>>
where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Provided Methods§

Source

fn execute_language_model_batch_workflow<'life0, 'life1, 'async_trait>( &'life0 mut self, model: LanguageModelType, expected_content_type: ExpectedContentType, input_tokens: &'life1 [Self::Seed], ) -> Pin<Box<dyn Future<Output = Result<(), E>> + Send + 'async_trait>>
where 'life0: 'async_trait, 'life1: 'async_trait, Self: Send + 'async_trait,

High-level method that ties it all together. Fixes the mismatch by enumerating chunk-slices properly and passing &[LanguageModelBatchAPIRequest] to process_batch_requests.

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors§