pub struct ContextWindowCompactionStrategy { /* private fields */ }Expand description
Token-budget compaction from a model’s context window (upstream
ContextWindowCompactionStrategy): the input budget is the window less
the output reservation; once the conversation passes
tool_eviction_threshold of it, older tool results are collapsed (the
last keep_last_tool_call_groups kept), and if it then still passes
truncation_threshold, the oldest messages are dropped down to the
eviction level.
Implementations§
Source§impl ContextWindowCompactionStrategy
impl ContextWindowCompactionStrategy
Sourcepub const DEFAULT_TOOL_EVICTION_THRESHOLD: f64 = 0.5
pub const DEFAULT_TOOL_EVICTION_THRESHOLD: f64 = 0.5
Upstream’s default eviction threshold.
Sourcepub const DEFAULT_TRUNCATION_THRESHOLD: f64 = 0.8
pub const DEFAULT_TRUNCATION_THRESHOLD: f64 = 0.8
Upstream’s default truncation threshold.
Sourcepub fn new(
max_context_window_tokens: usize,
max_output_tokens: usize,
) -> Result<ContextWindowCompactionStrategy, Error>
pub fn new( max_context_window_tokens: usize, max_output_tokens: usize, ) -> Result<ContextWindowCompactionStrategy, Error>
A strategy for a max_context_window_tokens window reserving
max_output_tokens for the reply.
Sourcepub fn with_tool_eviction_threshold(
self,
threshold: f64,
) -> Result<ContextWindowCompactionStrategy, Error>
pub fn with_tool_eviction_threshold( self, threshold: f64, ) -> Result<ContextWindowCompactionStrategy, Error>
The fraction of the input budget at which tool results are evicted.
Sourcepub fn with_truncation_threshold(
self,
threshold: f64,
) -> Result<ContextWindowCompactionStrategy, Error>
pub fn with_truncation_threshold( self, threshold: f64, ) -> Result<ContextWindowCompactionStrategy, Error>
The fraction of the input budget at which messages are truncated.
Sourcepub fn with_keep_last_tool_call_groups(
self,
keep: usize,
) -> ContextWindowCompactionStrategy
pub fn with_keep_last_tool_call_groups( self, keep: usize, ) -> ContextWindowCompactionStrategy
How many recent tool results eviction keeps (default 4).
Sourcepub fn preserve_first_user(self) -> ContextWindowCompactionStrategy
pub fn preserve_first_user(self) -> ContextWindowCompactionStrategy
Keep the earliest user message through truncation.
Sourcepub fn max_context_window_tokens(&self) -> usize
pub fn max_context_window_tokens(&self) -> usize
The model’s context window.
Sourcepub fn max_output_tokens(&self) -> usize
pub fn max_output_tokens(&self) -> usize
The output reservation.
Sourcepub fn input_budget_tokens(&self) -> usize
pub fn input_budget_tokens(&self) -> usize
The input budget: the window less the output reservation.
Sourcepub fn tool_eviction_threshold(&self) -> f64
pub fn tool_eviction_threshold(&self) -> f64
The eviction threshold.
Sourcepub fn truncation_threshold(&self) -> f64
pub fn truncation_threshold(&self) -> f64
The truncation threshold.
Trait Implementations§
Source§impl Clone for ContextWindowCompactionStrategy
impl Clone for ContextWindowCompactionStrategy
Source§fn clone(&self) -> ContextWindowCompactionStrategy
fn clone(&self) -> ContextWindowCompactionStrategy
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more