Struct Options

Source

pub struct Options(/* private fields */);

Expand description

Triton server creation options.

Implementations§

Source §

impl Options

Source

pub fn new<P: AsRef<Path>>(repository: P) -> Result<Self, Error>

Create a new server options object.
The path must be the full absolute path to the model repository.
This function can be called multiple times with different paths to set multiple model repositories.
Note that if a model is not unique across all model repositories at any time, the model will not be available.

Source

pub fn server_id<I: AsRef<str>>(&mut self, id: I) -> Result<&mut Self, Error>

Set the textual ID for the server in a server options. The ID is a name that identifies the server.

Source

pub fn startup_model<S: AsRef<str>>( &mut self, model: S, ) -> Result<&mut Self, Error>

Set the model to be loaded at startup in a server options.
The model must be present in one, and only one, of the specified model repositories.
This function can be called multiple times with different model name to set multiple startup models.
Note that it only takes affect with Control::Explicit set.

Source

pub fn model_control_mode(&mut self, mode: Control) -> Result<&mut Self, Error>

Set the model control mode in a server options. For each mode the models will be managed as the following:

Control::None: the models in model repository will be loaded on startup. After startup any changes to the model repository will be ignored. Calling poll_model_repository will result in an error.

Control::Poll: the models in model repository will be loaded on startup. The model repository can be polled periodically using poll_model_repository and the server will load, unload, and updated models according to changes in the model repository.

Control::Explicit: the models in model repository will not be loaded on startup. The corresponding model control APIs must be called to load / unload a model in the model repository.

Source

pub fn strict_model_config(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable strict model configuration handling in a server options.

Source

pub fn model_config_name<C: AsRef<str>>( &mut self, config_name: C, ) -> Result<&mut Self, Error>

Set the custom model configuration name to load for all models. Fall back to default config file if empty.

config_name The name of the config file to load for all models.

Source

pub fn rate_limiter_mode(&mut self, mode: Limit) -> Result<&mut Self, Error>

Set the rate limit mode.

Limit::ExecCount: The rate limiting prioritizes the inference execution using the number of times each instance has got a chance to run. The execution gets to run only when its resource constraints are satisfied.

Limit::Off: The rate limiting is turned off and the inference gets executed whenever an instance is available.

By default, execution count is used to determine the priorities.

Source

pub fn add_rate_limiter_resource<N: AsRef<str>>( &mut self, name: N, count: u64, device: i32, ) -> Result<&mut Self, Error>

Add resource count for rate limiting.
name: The name of the resource.
count: The count of the resource.
device: The device identifier for the resource. A value of -1 indicates that the specified number of resources are available on every device.

The device value is ignored for a global resource.
The server will use the rate limiter configuration specified for instance groups in model config to determine whether resource is global.
In case of conflicting resource type in different model configurations, server will raise an appropriate error while loading model.

Source

pub fn pinned_memory_pool_byte_size( &mut self, size: u64, ) -> Result<&mut Self, Error>

Set the total pinned memory byte size that the server can allocate .
The pinned memory pool will be shared across Triton itself and the backends that use MemoryManager to allocate memory.
size: The pinned memory pool byte size.

Source

pub fn cuda_memory_pool_byte_size( &mut self, device: i32, size: u64, ) -> Result<&mut Self, Error>

Set the total CUDA memory byte size that the server can allocate on given GPU device.
The pinned memory pool will be shared across Triton itself and the backends that use MemoryManager to allocate memory.
device: The GPU device to allocate the memory pool.
size: The pinned memory pool byte size.

Source

pub fn cuda_virtual_address_size( &mut self, device: i32, size: usize, ) -> Result<&mut Self, Error>

Set the size of the virtual address space that will be used for growable memory in implicit state.

gpu_device The GPU device to set the CUDA virtual address space size
`size`` The size of the CUDA virtual address space.

Source

pub fn response_cache_byte_size( &mut self, size: u64, ) -> Result<&mut Self, Error>

👎Deprecated

Deprecated. See Options::response_cache_config instead.
Set the total response cache byte size that the server can allocate in CPU memory.
The response cache will be shared across all inference requests and across all models.
size: The total response cache byte size.

Source

pub fn response_cache_directory<P: AsRef<Path>>( &mut self, cache_dir: P, ) -> Result<&mut Self, Error>

Set the directory containing cache shared libraries. This directory is searched when looking for cache implementations.

cache_dir The full path of the cache directory.

Source

pub fn response_cache_config<N: AsRef<str>, J: AsRef<str>>( &mut self, cache_name: N, config_json: J, ) -> Result<&mut Self, Error>

Set the cache config that will be used to initialize the cache implementation for `cache_name``.

It is expected that the cache_name`` provided matches a directory inside the cache_dirused for [Options::response_cache_directory]. The defaultcache_diris "/opt/tritonserver/caches", so for acache_name` of “local”, Triton would expect to find the “local” cache implementation at “/opt/tritonserver/caches/local/libtritoncache_local.so”

Altogether an example for the “local” cache implementation would look like:

let cache_name = "local";
let config_json = "({\"size\": 1048576})"
options.response_cache_config(cache_name, config_json)?;

cache_name The name of the cache. Example names would be “local”, “redis”, or the name of a custom cache implementation.
config_json The string representation of config JSON that is used to initialize the cache implementation.

Source

pub fn min_supported_compute_capability( &mut self, capability: f64, ) -> Result<&mut Self, Error>

Set the minimum support CUDA compute capability.
capability: The minimum CUDA compute capability.

Source

pub fn exit_on_error(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable exit-on-error. True to enable exiting on initialization error, false to continue.

Source

pub fn strict_readiness(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable strict readiness handling.

Source

pub fn exit_timeout(&mut self, timeout: Duration) -> Result<&mut Self, Error>

Set the exit timeout.

Source

pub fn buffer_manager_thread_count( &mut self, thread: usize, ) -> Result<&mut Self, Error>

Set the number of threads used in buffer manager.

Source

pub fn model_load_thread_count( &mut self, thread_count: usize, ) -> Result<&mut Self, Error>

Set the number of threads to concurrently load models in a server options.

thread_count The number of threads.

Source

pub fn model_retry_count( &mut self, retry_count: usize, ) -> Result<&mut Self, Error>

Set the number of retry to load a model in a server options.

retry_count The number of retry.

Source

pub fn peer_access( &mut self, enable_peer_access: bool, ) -> Result<&mut Self, Error>

Enable peer access to allow GPU device to directly access the memory of another GPU device. Note that even when this option is set to True, Triton will only try to enable peer access and might fail to enable it if the underlying system doesn’t support peer access.

enable_peer_access Whether to enable peer access or not.

Source

pub fn model_namespacing( &mut self, enable_namespace: bool, ) -> Result<&mut Self, Error>

Enable model namespacing to allow serving models with the same name if they are in different namespaces.

enable_namespace Whether to enable model namespacing or not.

Source

pub fn log_file<P: AsRef<str>>( &mut self, log_file: P, ) -> Result<&mut Self, Error>

Provide a log output file.

log_file a string defining the file where the log outputs will be saved. An empty string for the file name will cause triton to direct logging facilities to the console.

Source

pub fn log_info(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable info level logging.

Source

pub fn log_warn(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable warning level logging.

Source

pub fn log_error(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable error level logging.

Source

pub fn log_format(&mut self, log_format: LogFormat) -> Result<&mut Self, Error>

Source

pub fn log_verbose(&mut self, level: i32) -> Result<&mut Self, Error>

Set verbose logging level. Level zero disables verbose logging.

Source

pub fn metrics(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable metrics collection in a server options.

Source

pub fn gpu_metrics(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable GPU metrics collection in a server options. GPU metrics are collected if both this option and Options::metrics are set.

Source

pub fn cpu_metrics(&mut self, enable: bool) -> Result<&mut Self, Error>

Enable or disable CPU metrics collection in a server options. CPU metrics are collected if both this option and Options::metrics are true. True to enable CPU metrics, false to disable.

Source

pub fn metrics_interval( &mut self, interval: Duration, ) -> Result<&mut Self, Error>

Set the interval for metrics collection in a server options. This is 2000 milliseconds by default.

Source

pub fn backend_directory<P: AsRef<Path>>( &mut self, path: P, ) -> Result<&mut Self, Error>

Set the directory containing backend shared libraries.
This directory is searched last after the version and model directory in the model repository when looking for the backend shared library for a model.
If the backend is named ‘be’ the directory searched is ‘backend_dir’/be/libtriton_be.so.

Source

pub fn repo_agent_directory<P: AsRef<Path>>( &mut self, path: P, ) -> Result<&mut Self, Error>

Set the directory containing repository agent shared libraries.
This directory is searched when looking for the repository agent shared library for a model.
If the backend is named ‘ra’ the directory searched is ‘repoagent_dir’/ra/libtritonrepoagent_ra.so.

Source

pub fn model_load_device_limit( &mut self, kind: InstanceGroupKind, device: i32, fraction: f64, ) -> Result<&mut Self, Error>

Specify the limit on memory usage as a fraction on the device identified by ‘kind’ and ‘device_id’. If model loading on the device is requested and the current memory usage exceeds the limit, the load will be rejected. If not specified, the limit will not be set.

Currently support InstanceGroupKind::Gpu

kind The kind of the device.
device The id of the device.
fraction The limit on memory usage as a fraction.

Source

pub fn backend_config<N, S, V>( &mut self, name: N, setting: S, value: V, ) -> Result<&mut Self, Error>
where N: AsRef<str>, S: AsRef<str>, V: AsRef<str>,

Set a configuration setting for a named backend in a server options.
name: The name of the backend.
setting: The name of the setting.
value: The setting value.

Source

pub fn host_policy<N, S, V>( &mut self, name: N, setting: S, value: V, ) -> Result<&mut Self, Error>
where N: AsRef<str>, S: AsRef<str>, V: AsRef<str>,

Set a host policy setting for a given policy name in a server options.
name: The name of the policy.
setting: The name of the setting.
value: The setting value.

Source

pub fn metrics_config<N, S, V>( &mut self, name: N, setting: S, value: V, ) -> Result<&mut Self, Error>
where N: AsRef<str>, S: AsRef<str>, V: AsRef<str>,

Set a configuration setting for metrics in server options.

name The name of the configuration group. An empty string indicates a global configuration option.
setting The name of the setting.
value The setting value.