Expand description
§inference-runtime-mistralrs
mistralrs runner for atomr-infer. Wraps mistralrs::Model +
mistralrs::TextModelBuilder behind the ModelRunner trait so
Mistral.rs participates in the same Deployment actor topology as
the OpenAI / Anthropic / vLLM / TensorRT runners. Doc §10.3.
The model is loaded lazily on the first call to
ModelRunner::execute (mistralrs’s builder downloads from
HuggingFace, which can take minutes for 7B+ models — eager loading
would block the runner’s constructor for too long).
Default-features-off the crate compiles to a typed-error stub;
cargo build --features remote-only therefore pulls no candle /
cuda dependencies via this crate.
§MSRV note
mistralrs 0.8 declares MSRV 1.88. The atomr-infer workspace MSRV is 1.78 for remote-only builds; operators enabling this runner need a toolchain that satisfies mistralrs’s own MSRV.