Struct llama_cpp::CompletionHandle

pub struct CompletionHandle { /* private fields */ }

A handle (and channel) to an ongoing completion job on an off thread.

If this structure is dropped, the off thread is stopped.

Implementations

impl CompletionHandle

pub fn next_token(&self) -> Option<Token>

Blocks the current thread, resolving to the next completed token, or None if EOS is reached.

pub async fn next_token_async(&self) -> Option<Token>

Asynchronously yields the current thread, resolving to the next completed token, or None if EOS is reached.