Module scheduler 

Backend-agnostic batch scheduler for agent workloads.

This module provides a pipeline scheduler that groups work items by model, prefix hash, and context length for optimal cache utilization across different LLM backends (Anthropic Batch API, Ollama, etc.).

Architecture

The scheduler operates as a pipeline where batches are interleaved:

Batch 1 [agents A,B,C]: PERCEIVE → submit THINK → poll → ACT → submit REFLECT
Batch 2 [agents D,E,F]:             PERCEIVE → submit THINK → poll → ACT ...
  (D,E,F see A,B,C's committed actions in their perceptions)

This ensures agents in later batches observe earlier batches’ actions, creating a natural information flow without strict phase barriers.

Grouping

Work items are grouped by priority:

1. Model — most expensive to switch (weight loading / pricing)
2. Prefix hash — KV cache reuse on both Anthropic and Ollama
3. Context length — avoid memory reallocation on Ollama

Items waiting too long are promoted regardless of grouping optimality to prevent starvation.

Structs

BatchGroup

A group of work items that should be submitted together.

GroupingConfig

Configuration for the grouping algorithm.

Scheduler

Pipeline scheduler that manages batch submission and interleaving.

SchedulerConfig

Configuration for the pipeline scheduler.

WorkItem

A unit of work: one agent’s prompt for one cycle step.

WorkResult

Result of processing a single work item.

Enums

BatchError

Errors that can occur during batch processing.

BatchState

The state of a polled batch.

CycleStep

Identifies what step in the agent cycle a work item represents.

Traits

BatchBackend

Backend-agnostic interface for submitting and polling batch work.

PendingHandle

A handle to a submitted batch, returned by BatchBackend::submit.

Functions

group_work_items

Group work items into BatchGroups by (model, prefix_hash, context_bucket).