Struct SequenceState 

pub struct SequenceState {

    pub request_id: RequestId,
    pub original_request: InferenceRequest,
    pub input_tokens: Vec<TokenId>,
    pub generated_tokens: Vec<TokenId>,
    pub phase: RequestPhase,
    pub rng: SamplingRng,
    pub prefill_complete: bool,
    pub prefill_tokens_processed: usize,
    pub stream_sender: Option<Sender<Result<StreamChunk>>>,
    pub response_sender: Option<Sender<Result<InferenceResponse>>>,
    pub start_time: Instant,
    pub first_emit_at: Option<Instant>,
    pub last_emit_at: Option<Instant>,
    pub emitted_chunks: u32,
    pub tokens_this_iteration: usize,
    pub preemption_count: usize,
    pub structured_output_processor: Option<StructuredOutputProcessor>,
    pub stop_token_ids: HashSet<u32>,
    pub model_eos_token_ids: Vec<u32>,
    pub forbidden_token_ids: HashSet<u32>,
    pub initial_forbidden_token_ids: HashSet<u32>,
    pub tokenizer_base_vocab_size: Option<usize>,
    pub allowed_extended_token_ids: HashSet<u32>,
    pub stop_text_seqs: Vec<String>,
    pub argmax_token_mask: Option<TokenSelectionMask>,
    pub initial_argmax_token_mask: Option<TokenSelectionMask>,
    pub pending_decoded_utf8_fragment: bool,
    pub pending_decoded_utf8_bytes: Vec<u8>,
    pub streamed_text_len: usize,
    /* private fields */
}

State of a running sequence in the continuous batch.

Fields

request_id: RequestId

original_request: InferenceRequest

Original request — kept for re-submission after preemption.

input_tokens: Vec<TokenId>

generated_tokens: Vec<TokenId>

phase: RequestPhase

rng: SamplingRng

prefill_complete: bool

prefill_tokens_processed: usize

Number of prompt tokens already written into the model KV cache by opt-in unified chunked prefill. Zero for the normal full-prefill path.

stream_sender: Option<Sender<Result<StreamChunk>>>

response_sender: Option<Sender<Result<InferenceResponse>>>

start_time: Instant

first_emit_at: Option<Instant>

Wall-clock Instant at which the first SSE chunk was actually sent to the client stream. Populated lazily by send_stream_update the first time a non-empty delta is emitted (multi-byte UTF-8 buffering can defer that past the first scheduler-completed token). Used to record ferrum.engine.ttft_seconds and as the start point of the TPOT window.

last_emit_at: Option<Instant>

Wall-clock Instant at the most recent successfully-sent chunk. Used to compute per-token ITL deltas (ferrum.engine.itl_seconds).

emitted_chunks: u32

Count of stream chunks successfully sent to the client. Lags generated_tokens.len() by the number of tokens currently buffered for a multi-byte UTF-8 sequence (so a Chinese char split across 2 BPE tokens emits once, increments the count by 1).

tokens_this_iteration: usize

preemption_count: usize

Number of times this request has been preempted.

structured_output_processor: Option<StructuredOutputProcessor>

Tokenizer-aware hard grammar for json_object and strict schema.

stop_token_ids: HashSet<u32>

Single-token stop ids: model’s EOS + any stop_sequences that encode to exactly one token. Checked against the last generated token each step — replaces the old “token id near top of vocab = EOS” placeholder. Built from tokenizer.eos_token, a common-EOS fallback list (</s>, <|im_end|>, <|endoftext|>, <|eot_id|>), and one-token encodings of sampling_params.stop_sequences.

model_eos_token_ids: Vec<u32>

Model-owned EOS ids, kept separate from user stop conditions so a response-completion boundary can delay only model termination.

forbidden_token_ids: HashSet<u32>

Token IDs that should never be sampled as normal output. Used for tokenizer/model vocab holes such as Qwen3’s reserved tail IDs and literal <unk / <unk> pieces.

initial_forbidden_token_ids: HashSet<u32>

Token IDs masked only before the first generated token.

tokenizer_base_vocab_size: Option<usize>

Base tokenizer vocabulary size. IDs above this are allowed only when they are explicitly whitelisted in allowed_extended_token_ids.

allowed_extended_token_ids: HashSet<u32>

stop_text_seqs: Vec<String>

Multi-token text stop sequences (stop_sequences entries that don’t resolve to a single token). Checked via accumulated decoded text.

argmax_token_mask: Option<TokenSelectionMask>

Base token-validity mask for model-side greedy argmax.

initial_argmax_token_mask: Option<TokenSelectionMask>

First-token variant that also applies initial_forbidden_token_ids.

pending_decoded_utf8_fragment: bool

A byte-level token may decode to a trailing replacement character until a later token supplies the remaining UTF-8 bytes. The next step needs full logits so the engine can reject candidates that would flush that incomplete fragment into user-visible output.

pending_decoded_utf8_bytes: Vec<u8>

Raw byte-level tokenizer suffix that is a syntactically valid but incomplete UTF-8 scalar. Unlike a lossy decoded string, this preserves whether a continuation token can actually complete the pending scalar.

streamed_text_len: usize

Bytes of decoded generated_tokens already flushed via the stream channel. Used by send_stream_update to compute per-call delta from the full-history decode, so multi-byte UTF-8 sequences (Chinese chars, emoji) that span several BPE tokens don’t get rendered as \u{FFFD} replacement chars when decoded one token at a time.

Implementations

impl SequenceState

pub fn new(request: InferenceRequest, input_tokens: Vec<TokenId>) -> Self

pub fn new_with_tokenizer( request: InferenceRequest, input_tokens: Vec<TokenId>, tokenizer: Option<Arc<dyn Tokenizer + Send + Sync>>, ) -> Self

Build sequence state, optionally wiring a tokenizer for constrained decoding. Test-only direct constructors build a local grammar factory; product entrypoints use the fallible shared-factory constructor below.

pub fn new_with_tokenizer_and_model_vocab_size( request: InferenceRequest, input_tokens: Vec<TokenId>, tokenizer: Option<Arc<dyn Tokenizer + Send + Sync>>, model_vocab_size: Option<usize>, ) -> Self

pub fn try_new_with_tokenizer_model_vocab_and_structured_factory( request: InferenceRequest, input_tokens: Vec<TokenId>, tokenizer: Option<Arc<dyn Tokenizer + Send + Sync>>, model_vocab_size: Option<usize>, shared_structured_factory: Option<&StructuredOutputFactory>, ) -> Result<Self>

pub fn total_tokens(&self) -> usize

pub fn sampling_params(&self) -> &SamplingParams

Original immutable sampling parameters used to prepare this request.

pub fn prefill_context_tokens(&self) -> Vec<TokenId>

pub fn prefill_context_len(&self) -> usize

pub fn model_decode_metadata(&self) -> HashMap<String, Value>

pub fn model_decode_logits_policy(&self) -> LogitsReturnPolicy

pub fn validate_and_commit_model_greedy_argmax_token( &mut self, tokenizer: Option<&(dyn Tokenizer + Send + Sync)>, token: TokenId, ) -> Result<()>

pub fn requires_engine_full_logits_for_sampling(&self) -> bool

pub fn has_structured_output_constraint(&self) -> bool

pub fn requires_full_logits_for_sampling(&self) -> bool

pub fn reset_guided_processors(&mut self) -> Result<()>

pub fn stop_reason( &self, tokenizer: Option<&(dyn Tokenizer + Send + Sync)>, ) -> Option<FinishReason>

Return the reason this sequence should stop, if any.

Checks: (1) last generated token is in the resolved stop_token_ids set (model EOS + any single-token stop_sequences), (2) decoded text contains a multi-token user stop sequence, (3) max-tokens budget is exhausted. Text-stop decoding only runs for requests that supplied a multi-token stop string, so the common EOS path stays cheap.

pub fn should_stop(&self) -> bool

Cheap stop check for tests and callers that do not have tokenizer access. Engine hot paths use stop_reason through EngineInner.

pub fn sample_and_commit_with_processors( &mut self, logits: &mut [f32], ) -> Result<TokenId>

Sample and commit the next token with the full processor chain.

pub fn sample_and_commit_with_processors_and_tokenizer( &mut self, logits: &mut [f32], tokenizer: Option<&(dyn Tokenizer + Send + Sync)>, ) -> Result<TokenId>

Trait Implementations

impl Debug for SequenceState

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for SequenceState

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.