Expand description
OpenAI-convention token accounting plus llama.cpp-style timings.
Counted from the exact token ids the generation loop processed (prompt after BOS insertion, and every generated id), not re-tokenized after the fact – re-tokenizing decoded text is not guaranteed to round-trip to the same count.
Why the server reports timings at all, when a client can hold a stopwatch: the client’s stopwatch measures the network, the proxy’s buffer and its own event loop. More importantly it cannot separate prefill from decode, and a UI that divides total tokens by total wall time reports a 50 tok/s model as 5 tok/s whenever the prompt is long. Every downstream number built on that is then wrong in the same direction. So the phases are reported separately and the client is never asked to infer one from the other.
Every timing is optional: a cached response, a batched decode, or an engine path that does not time itself must be able to answer honestly rather than emit a plausible zero.
Structs§
- Completion
Tokens Details - OpenAI’s
usage.completion_tokens_details. - Usage