Skip to main content

Module inference_usage

Module inference_usage 

Source
Expand description

What one provider call cost, recorded where it lands.

A run bills for four kinds of call and, until this module existed, counted one of them. Stage turns went into TokenTotals; the compaction, title, and routing lanes each threw their usage away at the outcome boundary, because each channel carried only the payload its collector wanted - a summary, a title, a stage name - and usage was not it.

Two things follow from putting the accounting in one place rather than repeating it per lane. The cumulative totals finally cover every call, so a run’s reported spend is what the provider actually billed. And each call writes a RunRecord::InferenceUsage as it lands, which is the part RunRecord::Progress cannot do: progress counters are cumulative, so two calls between two ticks arrive as their sum, and a chart of that sum shows a spike no single call ever made.

Structs§

CallUsage
Everything one call needs to report itself.

Functions§

record_call
Fold one call into the run’s cumulative totals and journal what it cost.