Skip to main content

ModelTokenMeter

orchestral_runtime

Trait ModelTokenMeter 

Source 
pub trait ModelTokenMeter: Send + Sync {
    // Required methods
    fn meter_descriptor(&self) -> ModelTokenMeterDescriptor;
    fn count_request_input(
        &self,
        messages: &[ModelMessage],
        tools: &[ModelToolDefinition],
    ) -> Result<u64, ModelError>;

    // Provided methods
    fn supports_observed_prefix_estimation(&self) -> bool { ... }
    fn estimate_context_input(
        &self,
        messages: &[ModelMessage],
        tools: &[ModelToolDefinition],
    ) -> Result<ModelContextEstimate, ModelError> { ... }
}
Expand description

Model-family token accounting boundary. A production adapter must provide either its exact tokenizer or a documented conservative upper bound over the provider-specific serialized request input.

Required Methods§

Source

fn meter_descriptor(&self) -> ModelTokenMeterDescriptor

Source

fn count_request_input( &self, messages: &[ModelMessage], tools: &[ModelToolDefinition], ) -> Result<u64, ModelError>

Provided Methods§

Source

fn supports_observed_prefix_estimation(&self) -> bool

Whether differences between this meter’s soft estimates may be added to observed input usage for an unchanged request prefix. This remains estimated accounting, never a hard bound. Opting in must change the meter descriptor’s immutable identity; unknown meters retain full-input estimation. The runtime separately verifies the prefix and Run scope.

Source

fn estimate_context_input( &self, messages: &[ModelMessage], tools: &[ModelToolDefinition], ) -> Result<ModelContextEstimate, ModelError>

Estimate space for context selection when no hard input/cost ceiling applies. The default retains the existing certified count. Adapters overriding this method must bind its algorithm into their descriptor’s immutable identity. Estimates must not replace count_request_input for resource reservations or validation of observed usage.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§