pub struct Inference { /* private fields */ }Expand description
Serves rlm.infer, rlm.load and rlm.frames for one lane; everything else goes to inner.
Implementations§
Source§impl Inference
impl Inference
Sourcepub fn new(
inner: Arc<dyn HostHandler>,
model: Model,
api_key: Option<String>,
transport: Arc<dyn Transport>,
context_dir: impl Into<PathBuf>,
) -> Self
pub fn new( inner: Arc<dyn HostHandler>, model: Model, api_key: Option<String>, transport: Arc<dyn Transport>, context_dir: impl Into<PathBuf>, ) -> Self
Frames call model through transport; loaded handles live in context_dir (created on first load). The
frames are this handler’s own until Inference::with_session shares a session’s.
Sourcepub fn with_session(self, session: Arc<FrameSession>, lane: &str) -> Self
pub fn with_session(self, session: Arc<FrameSession>, lane: &str) -> Self
Serve lane’s requests over the session’s frames (its ledger, traces, running frames and task host).
Sourcepub fn with_models(self, models: ModelResolver) -> Self
pub fn with_models(self, models: ModelResolver) -> Self
Frames may name other models (model=, rlm.frameModel): models resolves each to its model and key.
Sourcepub fn with_stream(self, stream: Arc<dyn StreamFn>) -> Self
pub fn with_stream(self, stream: Arc<dyn StreamFn>) -> Self
Frame requests go through stream (the session’s request stack) instead of this handler’s transport.
Trait Implementations§
Auto Trait Implementations§
impl !RefUnwindSafe for Inference
impl !UnwindSafe for Inference
impl Forget for Inference
impl Freeze for Inference
impl Move for Inference
impl Send for Inference
impl Sync for Inference
impl Unpin for Inference
impl UnsafeUnpin for Inference
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more