Struct llama_cpp::CompletionHandle

pub struct CompletionHandle<'a> { /* private fields */ }

A handle (and channel) to an ongoing completion job on an off thread.

If this structure is dropped, the off thread is stopped.

Implementations

impl<'a> CompletionHandle<'a>

pub fn next_token(&mut self) -> Option<CompletionToken<'_>>

Blocks the current thread, resolving to the next completed token, or None if EOS is reached.

pub async fn next_token_async(&mut self) -> Option<CompletionToken<'_>>

Asynchronously yields the current thread, resolving to the next completed token, or None if EOS is reached.