pub struct Server { /* private fields */ }Expand description
Inference server object.
Implementations§
Source§impl Server
impl Server
Sourcepub fn stop(&self) -> Result<(), Error>
pub fn stop(&self) -> Result<(), Error>
Stop a server object. A server can’t be restarted once it has been stopped.
Sourcepub fn create_request<M: AsRef<str>>(
&self,
model: M,
version: i64,
) -> Result<Request<'_>, Error>
pub fn create_request<M: AsRef<str>>( &self, model: M, version: i64, ) -> Result<Request<'_>, Error>
Create a request to the model model of version version.
If version is set as -1, the server will choose a version based on the model’s policy.
Sourcepub fn poll_model_repository(&mut self) -> Result<(), Error>
pub fn poll_model_repository(&mut self) -> Result<(), Error>
Check the model repository for changes and update server state based on those changes.
Sourcepub fn set_exit_timeout(
&mut self,
timeout: Duration,
) -> Result<&mut Self, Error>
pub fn set_exit_timeout( &mut self, timeout: Duration, ) -> Result<&mut Self, Error>
Set the exit timeout on the server object. This value overrides the value initially set through server options and provides a mechanism to update the exit timeout while the serving is running.
timeout The exit timeout.
Sourcepub fn register_model_repo<P: AsRef<Path>, N: AsRef<str>>(
&mut self,
repository: P,
name_mapping: HashMap<String, String>,
) -> Result<&mut Self, Error>
pub fn register_model_repo<P: AsRef<Path>, N: AsRef<str>>( &mut self, repository: P, name_mapping: HashMap<String, String>, ) -> Result<&mut Self, Error>
Register a new model repository. Not available in polling mode.
repository The full path to the model repository.
name_mapping List of name_mapping parameters.
Each mapping has the model directory name as its key,
overridden model name as its value.
Sourcepub fn unregister_model_repo<P: AsRef<Path>, N: AsRef<str>>(
&mut self,
repository: P,
) -> Result<&mut Self, Error>
pub fn unregister_model_repo<P: AsRef<Path>, N: AsRef<str>>( &mut self, repository: P, ) -> Result<&mut Self, Error>
Unregister a model repository. Not available in polling mode.
repository_path The full path to the model repository.
Sourcepub fn is_ready(&self) -> Result<bool, Error>
pub fn is_ready(&self) -> Result<bool, Error>
Returns true if server is ready, false otherwise.
Sourcepub fn model_is_ready<N: AsRef<str>>(
&self,
name: N,
version: i64,
) -> Result<bool, Error>
pub fn model_is_ready<N: AsRef<str>>( &self, name: N, version: i64, ) -> Result<bool, Error>
Returns true if the model is ready.
name: The name of the model to get readiness for.
version: The version of the model to get readiness for. If -1 then the server will choose a version based on the model’s policy. \
Sourcepub fn model_batch_properties<N: AsRef<str>>(
&self,
name: N,
version: i64,
) -> Result<Batch, Error>
pub fn model_batch_properties<N: AsRef<str>>( &self, name: N, version: i64, ) -> Result<Batch, Error>
Get the batch properties of the model.
name: The name of the model.
version: The version of the model. If -1 then the server will choose a version based on the model’s policy. \
Sourcepub fn model_transaction_properties<N: AsRef<str>>(
&self,
name: N,
version: i64,
) -> Result<Transaction, Error>
pub fn model_transaction_properties<N: AsRef<str>>( &self, name: N, version: i64, ) -> Result<Transaction, Error>
Get the transaction policy of the model.
name: The name of the model.
version: The version of the model. If -1 then the server will choose a version based on the model’s policy. \
Sourcepub fn metadata(&self) -> Result<Server, Error>
pub fn metadata(&self) -> Result<Server, Error>
Get the metadata of the server as a Message(json) object.
Sourcepub fn model_metadata<N: AsRef<str>>(
&self,
name: N,
version: i64,
) -> Result<Model, Error>
pub fn model_metadata<N: AsRef<str>>( &self, name: N, version: i64, ) -> Result<Model, Error>
Get the metadata of a model as a Message(json) object.
name: The name of the model.
version: The version of the model. If -1 then the server will choose a version based on the model’s policy.
Sourcepub fn model_statistics<N: AsRef<str>>(
&self,
name: N,
version: i64,
) -> Result<Value, Error>
pub fn model_statistics<N: AsRef<str>>( &self, name: N, version: i64, ) -> Result<Value, Error>
Get the statistics of a model as a Message(json) object.
name: The name of the model.
version: The version of the model. If -1 then the server will choose a version based on the model’s policy.
Sourcepub fn model_config<N: AsRef<str>>(
&self,
name: N,
version: i64,
config: u32,
) -> Result<Value, Error>
pub fn model_config<N: AsRef<str>>( &self, name: N, version: i64, config: u32, ) -> Result<Value, Error>
Get the configuration of a model as a Message(json) object.
name: The name of the model.
version: The version of the model. If -1 then the server will choose a version based on the model’s policy.
config: The model configuration will be returned in a format matching this version.
If the configuration cannot be represented in the requested version’s format then an error will be returned.
Currently only version 1 is supported.
Sourcepub fn model_index(&self, flags: State) -> Result<Vec<Index>, Error>
pub fn model_index(&self, flags: State) -> Result<Vec<Index>, Error>
Get the index of all unique models in the model repositories as a Message(json) object.
Sourcepub fn load_model<N: AsRef<str>>(&mut self, name: N) -> Result<(), Error>
pub fn load_model<N: AsRef<str>>(&mut self, name: N) -> Result<(), Error>
Load the requested model or reload the model if it is already loaded.
The function does not return until the model is loaded or fails to load .
name: The name of the model.
Sourcepub fn load_model_with_parametrs<N: AsRef<str>, P: AsRef<[Parameter]>>(
&mut self,
name: N,
parameters: P,
) -> Result<(), Error>
pub fn load_model_with_parametrs<N: AsRef<str>, P: AsRef<[Parameter]>>( &mut self, name: N, parameters: P, ) -> Result<(), Error>
Load the requested model or reload the model if it is already loaded, with load parameters provided.
The function does not return until the model is loaded or fails to load.
Currently the below parameter names are recognized:
- “config” : string parameter that contains a JSON representation of the model configuration. This config will be used for loading the model instead of the one in the model directory.
Can be usefull if is needed to load the model with altered config. For example, if it’s required to load only one exact version of the model (see Parameter::from_config_with_exact_version for more info).
name: The name of the model.
parameters: slice of parameters.
Sourcepub fn unload_model<N: AsRef<str>>(&mut self, name: N) -> Result<(), Error>
pub fn unload_model<N: AsRef<str>>(&mut self, name: N) -> Result<(), Error>
Unload the requested model.
Unloading a model that is not loaded on server has no affect and success code will be returned.
The function does not wait for the requested model to be fully unload and success code will be returned.
name: The name of the model.
Sourcepub fn unload_model_and_dependents<N: AsRef<str>>(
&mut self,
name: N,
) -> Result<(), Error>
pub fn unload_model_and_dependents<N: AsRef<str>>( &mut self, name: N, ) -> Result<(), Error>
Unload the requested model, and also unload any dependent model that was loaded along with the requested model
(for example, the models composing an ensemble).
Unloading a model that is not loaded on server has no affect and success code will be returned.
The function does not wait for the requested model and all dependent models to be fully unload and success code will be returned.
name: The name of the model.
Sourcepub fn metrics(&self) -> Result<PrometheusMetrics, Error>
pub fn metrics(&self) -> Result<PrometheusMetrics, Error>
Get the current Prometheus metrics for the server.