pub struct BatchTokenIDOutSlim {Show 22 fields
pub rids: Option<Vec<String>>,
pub output_ids: Vec<Vec<i32>>,
pub finished_reasons: Vec<String>,
pub prompt_tokens: Vec<i64>,
pub completion_tokens: Vec<i64>,
pub cached_tokens: Vec<i64>,
pub output_token_logprobs_val: Vec<Vec<f64>>,
pub output_token_logprobs_idx: Vec<Vec<i32>>,
pub engine_index: i64,
pub num_running: i64,
pub num_waiting: i64,
pub kv_active_pages: i64,
pub kv_total_pages: i64,
pub finish_reason_details: Vec<Option<FinishDetails>>,
pub spec_verify_ct: Vec<i64>,
pub accept_draft_tokens: Vec<Option<f64>>,
pub input_token_logprobs_val: Vec<Vec<Option<f64>>>,
pub input_token_logprobs_idx: Vec<Vec<i32>>,
pub input_top_logprobs_val: Vec<Vec<Option<Vec<f64>>>>,
pub input_top_logprobs_idx: Vec<Vec<Option<Vec<i32>>>>,
pub output_top_logprobs_val: Vec<Vec<Vec<f64>>>,
pub output_top_logprobs_idx: Vec<Vec<Vec<i32>>>,
}Expand description
One step’s output for a frontend that detokenizes itself.
Every column is parallel to rids. The load fields at the tail are
the scheduler’s latest snapshot; an older producer omits them and
they decode as zero.
Fields§
§rids: Option<Vec<String>>Request ids. Always present on this wire.
output_ids: Vec<Vec<i32>>Token ids generated this step, per request.
finished_reasons: Vec<String>"" while running, else the finish type ("stop", "length",
"abort").
prompt_tokens: Vec<i64>Prompt length.
completion_tokens: Vec<i64>Tokens generated so far.
cached_tokens: Vec<i64>Prompt tokens served from the prefix cache.
output_token_logprobs_val: Vec<Vec<f64>>Logprobs of this step’s tokens; empty for requests without logprobs.
output_token_logprobs_idx: Vec<Vec<i32>>The token ids output_token_logprobs_val refers to.
engine_index: i64The producing DP rank’s engine index.
num_running: i64Running requests.
num_waiting: i64Waiting requests.
kv_active_pages: i64KV pages in use by active requests.
kv_total_pages: i64KV pages in total.
finish_reason_details: Vec<Option<FinishDetails>>The finish reason in full (BaseFinishReason.to_json(): the
matched token, an abort’s message and codes), None while
running; one entry per request.
spec_verify_ct: Vec<i64>With speculative decoding, each request’s verify rounds so far; empty without.
accept_draft_tokens: Vec<Option<f64>>With speculative decoding, each finished request’s accepted
tokens per verify round, None before it finishes; empty
without.
input_token_logprobs_val: Vec<Vec<Option<f64>>>Prompt logprobs, shipped once per request: None for the first
position, then each position’s logprob given its prefix. This and
the top-k columns hold one entry per request in a frame where
some request carries one, and are empty for the whole frame
otherwise.
input_token_logprobs_idx: Vec<Vec<i32>>The prompt tokens input_token_logprobs_val refers to.
input_top_logprobs_val: Vec<Vec<Option<Vec<f64>>>>The top-k alternatives’ logprobs at each prompt position (None
for the first), most likely first.
input_top_logprobs_idx: Vec<Vec<Option<Vec<i32>>>>Their token ids.
output_top_logprobs_val: Vec<Vec<Vec<f64>>>The top-k alternatives’ logprobs of each token this step.
output_top_logprobs_idx: Vec<Vec<Vec<i32>>>Their token ids.
Implementations§
Source§impl BatchTokenIDOutSlim
impl BatchTokenIDOutSlim
Sourcepub fn requests(&self) -> Result<impl Iterator<Item = RequestOutput<'_>>>
pub fn requests(&self) -> Result<impl Iterator<Item = RequestOutput<'_>>>
The per-request rows, after checking that every column has one entry per request.