Struct Options

Source

pub struct Options(/* private fields */);

Expand description

Triton server creation options.

Implementations§

Source §

impl Options

Source

pub fn new<P: AsRef<Path>>(repository: P) -> Result<Self, Error>

Create a new server options object.
The path must be the full absolute path to the model repository.
This function can be called multiple times with different paths to set multiple model repositories.
Note that if a model is not unique across all model repositories at any time, the model will not be available.

Source

pub fn server_id<I: AsRef<str>>(&mut self, id: I) -> Result<&mut Self, Error>

Set the textual ID for the server in a server options. The ID is a name that identifies the server.

Source

pub fn startup_model<S: AsRef<str>>( &mut self, model: S, ) -> Result<&mut Self, Error>

Set the model to be loaded at startup in a server options.
The model must be present in one, and only one, of the specified model repositories.
This function can be called multiple times with different model name to set multiple startup models.
Note that it only takes affect with Control::EXPLICIT set.

Source

pub fn model_control_mode(&mut self, mode: Control) -> Result<&mut Self, Error>

Set the model control mode in a server options. For each mode the models will be managed as the following:

Control::NONE: the models in model repository will be loaded on startup. After startup any changes to the model repository will be ignored. Calling poll_model_repository will result in an error.

Control::POLL: the models in model repository will be loaded on startup. The model repository can be polled periodically using poll_model_repository and the server will load, unload, and updated models according to changes in the model repository.

Control::EXPLICIT: the models in model repository will not be loaded on startup. The corresponding model control APIs must be called to load / unload a model in the model repository.

Source

pub fn strict_model_config(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable strict model configuration handling in a server options.

Source

pub fn rate_limiter_mode(&mut self, mode: Limit) -> Result<&mut Self, Error>

Set the rate limit mode.

Limit::EXEC_COUNT: The rate limiting prioritizes the inference execution using the number of times each instance has got a chance to run. The execution gets to run only when its resource constraints are satisfied.

Limit::OFF: The rate limiting is turned off and the inference gets executed whenever an instance is available.

By default, execution count is used to determine the priorities.

Source

pub fn add_rate_limiter_resource<N: AsRef<str>>( &mut self, name: N, count: u64, device: i32, ) -> Result<&mut Self, Error>

Add resource count for rate limiting.
name: The name of the resource.
count: The count of the resource.
device: The device identifier for the resource. A value of -1 indicates that the specified number of resources are available on every device.

The device value is ignored for a global resource.
The server will use the rate limiter configuration specified for instance groups in model config to determine whether resource is global.
In case of conflicting resource type in different model configurations, server will raise an appropriate error while loading model.

Source

pub fn pinned_memory_pool_byte_size( &mut self, size: u64, ) -> Result<&mut Self, Error>

Set the total pinned memory byte size that the server can allocate .
The pinned memory pool will be shared across Triton itself and the backends that use MemoryManager to allocate memory.
size: The pinned memory pool byte size.

Source

pub fn cuda_memory_pool_byte_size( &mut self, device: i32, size: u64, ) -> Result<&mut Self, Error>

Set the total CUDA memory byte size that the server can allocate on given GPU device.
The pinned memory pool will be shared across Triton itself and the backends that use MemoryManager to allocate memory.
device: The GPU device to allocate the memory pool.
size: The pinned memory pool byte size.

Source

pub fn response_cache_byte_size( &mut self, size: u64, ) -> Result<&mut Self, Error>

Set the total response cache byte size that the server can allocate in CPU memory.
The response cache will be shared across all inference requests and across all models.
size: The total response cache byte size.

Source

pub fn min_supported_compute_capability( &mut self, capability: f64, ) -> Result<&mut Self, Error>

Set the minimum support CUDA compute capability.
capability: The minimum CUDA compute capability.

Source

pub fn exit_on_error(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable exit-on-error. True to enable exiting on initialization error, false to continue.

Source

pub fn strict_readiness(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable strict readiness handling.

Source

pub fn exit_timeout(&mut self, timeout: Duration) -> Result<&mut Self, Error>

Set the exit timeout.

Source

pub fn buffer_manager_thread_count( &mut self, thread: usize, ) -> Result<&mut Self, Error>

Set the number of threads used in buffer manager.

Source

pub fn log_info(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable info level logging.

Source

pub fn log_warn(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable warning level logging.

Source

pub fn log_error(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable error level logging.

Source

pub fn log_verbose(&mut self, level: i32) -> Result<&mut Self, Error>

Set verbose logging level. Level zero disables verbose logging.

Source

pub fn metrics(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable metrics collection in a server options.

Source

pub fn gpu_metrics(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable GPU metrics collection in a server options. GPU metrics are collected if both this option and Options::metrics are set.

Source

pub fn metrics_interval( &mut self, interval: Duration, ) -> Result<&mut Self, Error>

Set the interval for metrics collection in a server options. This is 2000 milliseconds by default.

Source

pub fn backend_directory<P: AsRef<Path>>( &mut self, path: P, ) -> Result<&mut Self, Error>

Set the directory containing backend shared libraries.
This directory is searched last after the version and model directory in the model repository when looking for the backend shared library for a model.
If the backend is named ‘be’ the directory searched is ‘backend_dir’/be/libtriton_be.so.

Source

pub fn repo_agent_directory<P: AsRef<Path>>( &mut self, path: P, ) -> Result<&mut Self, Error>

Set the directory containing repository agent shared libraries.
This directory is searched when looking for the repository agent shared library for a model.
If the backend is named ‘ra’ the directory searched is ‘repoagent_dir’/ra/libtritonrepoagent_ra.so.

Source

pub fn backend_config<N, S, V>( &mut self, name: N, setting: S, value: V, ) -> Result<&mut Self, Error>
where N: AsRef<str>, S: AsRef<str>, V: AsRef<str>,

Set a configuration setting for a named backend in a server options.
name: The name of the backend.
setting: The name of the setting.
value: The setting value.

Source

pub fn host_policy<N, S, V>( &mut self, name: N, setting: S, value: V, ) -> Result<&mut Self, Error>
where N: AsRef<str>, S: AsRef<str>, V: AsRef<str>,

Set a host policy setting for a given policy name in a server options.
name: The name of the policy.
setting: The name of the setting.
value: The setting value.