pub struct ContinuousBatchScheduler<D: ModelEngine, T: ModelEngine> { /* private fields */ }Expand description
High-throughput continuous batch scheduler executing iteration-level speculative inference. Dynamically pools multiple concurrent requests into unified forward passes.
Implementations§
Source§impl<D: ModelEngine + 'static, T: ModelEngine + 'static> ContinuousBatchScheduler<D, T>
impl<D: ModelEngine + 'static, T: ModelEngine + 'static> ContinuousBatchScheduler<D, T>
pub fn new( draft_model: D, target_model: T, cache_manager: Arc<PagedKvCacheManager>, max_batch_size: usize, ) -> Self
Sourcepub fn submit(&self, request: QueuedRequest) -> Result<(), EngineError>
pub fn submit(&self, request: QueuedRequest) -> Result<(), EngineError>
Enqueues an inference request for continuous batch processing.
Sourcepub fn active_len(&self) -> usize
pub fn active_len(&self) -> usize
Number of requests currently active in the generation batch.
Sourcepub fn stats(&self) -> SchedulerStats
pub fn stats(&self) -> SchedulerStats
Retrieves current scheduler telemetry.
Sourcepub async fn step_iteration(&self) -> Result<usize, EngineError>
pub async fn step_iteration(&self) -> Result<usize, EngineError>
Executes one iteration of continuous batching across all currently scheduled sequences. Returns the total number of committed tokens produced across all sequences in this iteration.
Auto Trait Implementations§
impl<D, T> !RefUnwindSafe for ContinuousBatchScheduler<D, T>
impl<D, T> !UnwindSafe for ContinuousBatchScheduler<D, T>
impl<D, T> Forget for ContinuousBatchScheduler<D, T>
impl<D, T> Freeze for ContinuousBatchScheduler<D, T>
impl<D, T> Move for ContinuousBatchScheduler<D, T>
impl<D, T> Send for ContinuousBatchScheduler<D, T>
impl<D, T> Sync for ContinuousBatchScheduler<D, T>
impl<D, T> Unpin for ContinuousBatchScheduler<D, T>
impl<D, T> UnsafeUnpin for ContinuousBatchScheduler<D, T>
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more