Crate inference

Library wrapping the gRPC interface to NVIDIA’s Triton Inference Server

Exposes a Rust API for creating Triton clients and running ML inference

Modules

error

Custom error implementations for the inference crate

inference

model

Client for interfacing with the Triton inference server over gRPC

registry

Implements model registry operations for a TritonClient

server

Triton gRPC requests related to the server’s health and readiness to serve a model

system

Triton server gRPC requests related to the state of the hardware the server is running on

Structs

TritonClient

A gRPC client for the Triton inference server

TritonModelBase

Base configuration for a model in a Triton Inference Server Used to store the model’s parameters and to generate inference requests that can be submitted through a TritonClient

Traits

TritonModel

Common methods for a derived data producing model to implement TODO: Add more useful methods and factor more of the SimpleModel implementation into generics