Struct TimeSlicePolicy 

pub struct TimeSlicePolicy { /* private fields */ }

Drain-first scheduling policy with a proactive background scheduler.

This policy minimizes GPU time wasted on model switches by following two principles:

1. Never preempt a serving model. When a request arrives for a non-active model, the policy defers to the background scheduler rather than switching reactively. The only exception is the staleness bound, which forces a switch if any request has waited longer than max_wait.

2. Switch when idle. The background scheduler periodically checks all models’ queue depths. When the active model has completely drained its queue (no pending requests, no in-flight), the scheduler switches to the model with the most waiting requests.

This is equivalent to “serve everything from the active model’s queue, then switch to whoever has the most demand.” The scheduler’s global visibility into all queue depths prevents the pathological back-and-forth switching that reactive policies cause under interleaved or dominant workloads.

In simulation across 12 workload profiles at switch costs from 2s to 20s, this policy achieves 61-94% GPU serving time vs CostAware’s 40-81% and FIFO’s 33-79%, while also delivering 2-6x lower maximum wait times.

Implementations

impl TimeSlicePolicy

pub fn new( eviction: EvictionPolicy, request_timeout: Duration, min_active_duration: Duration, max_wait: Duration, _min_quantum: Duration, tick_interval: Duration, _model_names: Vec<String>, ) -> Self

Trait Implementations

impl SwitchPolicy for TimeSlicePolicy

fn on_pending_request<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 PolicyContext, ) -> Pin<Box<dyn Future<Output = PolicyDecision> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Called when a request arrives for an inactive model

fn prepare_switch<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 mut SwitchContext, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Called before switching. Can wait for in-flight drain.

fn eviction_policy(&self) -> EvictionPolicy

Default eviction policy for models that don’t specify one

fn request_timeout(&self) -> Duration

Request timeout

fn min_active_duration(&self) -> Duration

Minimum time a model must stay active before it can be put to sleep. Prevents rapid wake/sleep thrashing that can cause GPU page faults.

fn scheduler_interval(&self) -> Option<Duration>

If Some(interval), the switcher will spawn a background scheduler that calls [schedule_tick] every interval.

fn schedule_tick(&self, ctx: &ScheduleContext) -> Option<String>

Called periodically by the background scheduler. Returns the model name to switch to, or None to stay on the current model.

fn on_switch_complete(&self, _from: &str, _to: &str, _duration: Duration)

Called after a switch completes successfully with the measured duration. Policies can use this to track empirical switch costs.