Struct TokenBudgetBatcher 

pub struct TokenBudgetBatcher { /* private fields */ }

Batch input accumulator that respects a total token budget instead of a fixed item count.

Current approach: estimates token count as text.len() / 4 (chars per token approximation). A production deployment can inject a real tokeniser via TokenBudgetBatcher::with_token_fn.

Example

use inference::batch::TokenBudgetBatcher;

let mut batcher = TokenBudgetBatcher::new(2048);
batcher.push("short text".to_string());
batcher.push("another short text".to_string());
let batches = batcher.finish();
assert_eq!(batches.len(), 1);

Implementations

impl TokenBudgetBatcher

pub fn new(token_budget: usize) -> Self

Create a batcher with the given token budget.

Defaults to character-count estimation (len / 4). Use with_token_fn to inject a real tokeniser.

pub fn with_token_fn( self, f: impl Fn(&str) -> usize + Send + Sync + 'static, ) -> Self

Replace the default character-count estimator with a real token counter.

pub fn push(&mut self, text: String)

Add a text to the current batch.

If adding text would exceed the token budget, the current batch is flushed first and a new batch starting with text is begun.

pub fn push_all(&mut self, texts: impl IntoIterator<Item = String>)

Add multiple texts at once.

pub fn finish(&mut self) -> Vec<Vec<String>>

Flush any pending batch and return all accumulated batches.

The batcher is reset after this call and can be reused.

pub fn pending_count(&self) -> usize

Number of texts accumulated in the current (unflushed) batch.

pub fn pending_tokens(&self) -> usize

Total tokens accumulated in the current (unflushed) batch.