pub struct ChunkedPrefillScheduler { /* private fields */ }Expand description
Chunked prefill orchestrator.
Implementations§
Source§impl ChunkedPrefillScheduler
impl ChunkedPrefillScheduler
pub fn new( cache_manager: Arc<PagedKvCacheManager>, config: ChunkedPrefillConfig, ) -> Self
Sourcepub fn submit_prompt(&mut self, sequence_id: u64, prompt: Vec<TokenId>)
pub fn submit_prompt(&mut self, sequence_id: u64, prompt: Vec<TokenId>)
Submits a new prompt sequence for chunked prefill.
Sourcepub fn pending_count(&self) -> usize
pub fn pending_count(&self) -> usize
Number of sequences currently undergoing chunked prefill.
Sourcepub async fn step_prefill<M: ModelEngine>(
&mut self,
model: &mut M,
) -> Result<Option<(u64, usize, bool)>, EngineError>
pub async fn step_prefill<M: ModelEngine>( &mut self, model: &mut M, ) -> Result<Option<(u64, usize, bool)>, EngineError>
Executes one chunked prefill step for the front task using the target model. Returns the number of tokens prefilled in this step, and whether the sequence completed.
Sourcepub fn take_completed_sequences(&mut self) -> Vec<u64>
pub fn take_completed_sequences(&mut self) -> Vec<u64>
Drains IDs of sequences that finished their prefill stage and are ready for speculative decoding.
Auto Trait Implementations§
impl !RefUnwindSafe for ChunkedPrefillScheduler
impl !UnwindSafe for ChunkedPrefillScheduler
impl Forget for ChunkedPrefillScheduler
impl Freeze for ChunkedPrefillScheduler
impl Move for ChunkedPrefillScheduler
impl Send for ChunkedPrefillScheduler
impl Sync for ChunkedPrefillScheduler
impl Unpin for ChunkedPrefillScheduler
impl UnsafeUnpin for ChunkedPrefillScheduler
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more