pub struct Session { /* private fields */ }Expand description
A stateful inference session bound to a Model.
Session is the low-level, modality-agnostic entry point: submit a
Request of Items and receive a Response of Items. For
higher-level, task-specific ergonomics use ChatSession,
EmbeddingsSession, or AudioSession, which Deref to Session.
Cloning a Session produces another handle to the same underlying native
session (shared conversation state).
Implementations§
Source§impl Session
impl Session
Sourcepub async fn new(model: &Model) -> Result<Session, FoundryLocalError>
pub async fn new(model: &Model) -> Result<Session, FoundryLocalError>
Open a session on a loaded model.
The model must already be loaded (see Model::load). The session
inherits the model’s default generation parameters until overridden with
set_options or per-request options.
Sourcepub async fn process_request(
&self,
request: Request,
) -> Result<Response, FoundryLocalError>
pub async fn process_request( &self, request: Request, ) -> Result<Response, FoundryLocalError>
Process a request and return the complete response.
Runs on a blocking worker thread; the returned future resolves when generation finishes.
§Cancellation
The returned future is cancel-on-drop: if it is dropped before it
resolves — for example via tokio::time::timeout, tokio::select!, or
aborting the task — the in-flight native request is cancelled and
generation stops as soon as possible rather than running to completion on
the detached worker. This mirrors the drop-cancels-generation behaviour of
process_streaming_request.
Sourcepub fn process_streaming_request(&self, request: Request) -> ItemStream
pub fn process_streaming_request(&self, request: Request) -> ItemStream
Process a request, streaming each output Item as it is
produced.
The returned ItemStream yields items until generation completes;
dropping it cancels generation.
Sourcepub async fn set_options(
&self,
options: RequestOptions,
) -> Result<(), FoundryLocalError>
pub async fn set_options( &self, options: RequestOptions, ) -> Result<(), FoundryLocalError>
Apply session-scoped options that persist across subsequent requests.
Sourcepub fn create_input_queue(&self) -> Result<ItemQueue, FoundryLocalError>
pub fn create_input_queue(&self) -> Result<ItemQueue, FoundryLocalError>
Create an ItemQueue for streaming incremental input into a request.
Attach the returned queue to a Request via
Request::with_input_queue, then push items as they become available.