Trait BatchBackend 

pub trait BatchBackend<P, R>: Send + Sync {
    type Handle: PendingHandle;

    // Required methods
    async fn submit(&self, items: Vec<WorkItem<P>>) -> Result<Self::Handle>;
    async fn poll(
        &self,
        handle: Self::Handle,
    ) -> Result<BatchState<R, Self::Handle>>;
    async fn count_tokens(&self, prompt: &P) -> Result<Option<u32>>;
    fn backend_name(&self) -> &str;
}

Backend-agnostic interface for submitting and polling batch work.

Implementations wrap specific LLM backends (Anthropic Batch API, Ollama, etc.) and handle prompt submission, polling, and result collection.

The type parameters:

• P — the prompt type (e.g. misanthropic::Prompt<'static>)
• R — the response type (e.g. misanthropic::prompt::Message<'static>)

Required Associated Types

type Handle: PendingHandle

The handle type returned by submit, used for polling.

Required Methods

async fn submit(&self, items: Vec<WorkItem<P>>) -> Result<Self::Handle>

Submit a batch of work items. Returns a handle for polling.

async fn poll( &self, handle: Self::Handle, ) -> Result<BatchState<R, Self::Handle>>

Poll a pending batch. Returns BatchState::Pending if still processing, or BatchState::Ready with all results.

async fn count_tokens(&self, prompt: &P) -> Result<Option<u32>>

Count tokens for a prompt. Used for grouping decisions and cache eligibility checks.

Returns None if the backend doesn’t support token counting, in which case the scheduler will skip token-based grouping.

fn backend_name(&self) -> &str

Human-readable backend name for logging.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors