triton_rs

Trait Backend

Source 
pub trait Backend {
    // Required method
    fn model_instance_execute(
        model: Model,
        requests: &[Request],
    ) -> Result<(), Error>;

    // Provided methods
    fn initialize() -> Result<(), Error> { ... }
    fn finalize() -> Result<(), Error> { ... }
    fn model_instance_initialize() -> Result<(), Error> { ... }
    fn model_instance_finalize() -> Result<(), Error> { ... }
}

Required Methods§

Source

fn model_instance_execute( model: Model, requests: &[Request], ) -> Result<(), Error>

Execute a batch of one or more requests on a model instance. This function is required. Triton will not perform multiple simultaneous calls to this function for a given model ‘instance’; however, there may be simultaneous calls for different model instances (for the same or different models).

Corresponds to TRITONBACKEND_ModelInstanceExecute.

Provided Methods§

Source

fn initialize() -> Result<(), Error>

Initialize a backend. This function is optional, a backend is not required to implement it. This function is called once when a backend is loaded to allow the backend to initialize any state associated with the backend. A backend has a single state that is shared across all models that use the backend.

Corresponds to TRITONBACKEND_Initialize.

Source

fn finalize() -> Result<(), Error>

Finalize for a backend. This function is optional, a backend is not required to implement it. This function is called once, just before the backend is unloaded. All state associated with the backend should be freed and any threads created for the backend should be exited/joined before returning from this function. Corresponds to TRITONBACKEND_Finalize.

Source

fn model_instance_initialize() -> Result<(), Error>

Initialize for a model instance. This function is optional, a backend is not required to implement it. This function is called once when a model instance is created to allow the backend to initialize any state associated with the instance.

Corresponds to TRITONBACKEND_ModelInstanceInitialize.

Source

fn model_instance_finalize() -> Result<(), Error>

Finalize for a model instance. This function is optional, a backend is not required to implement it. This function is called once for an instance, just before the corresponding model is unloaded from Triton. All state associated with the instance should be freed and any threads created for the instance should be exited/joined before returning from this function.

Corresponds to TRITONBACKEND_ModelInstanceFinalize.

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors§