Struct TokenBlockSequence 

pub struct TokenBlockSequence { /* private fields */ }

Represents a sequence of tokens, segmented into fixed-size, hashed blocks.

This structure manages a series of completed TokenBlocks and one PartialTokenBlock for accumulating incoming tokens. It provides methods for appending tokens (append, extend), removing tokens (pop, truncate, unwind), and accessing sequence information.

Hashing incorporates an initial SaltHash to ensure uniqueness across different contexts (e.g., different models, PEFTs).

Key Hashes:

• BlockHash: Hash of tokens within a single block (seeded by SaltHash).
• SequenceHash: Hash combining the previous block’s SequenceHash and the current block’s BlockHash (also seeded by SaltHash).

Implementations

impl TokenBlockSequence

pub fn new(tokens: Tokens, block_size: u32, salt_hash: Option<SaltHash>) -> Self

Creates a new TokenBlockSequence from an initial set of tokens.

The tokens are split into blocks of block_size. Any remaining tokens form the initial current_block.

Arguments

• tokens - The initial Tokens for the sequence.
• block_size - The fixed size for each TokenBlock. Must be greater than 0.
• salt_hash - An optional SaltHash. Defaults to 0 if None.

Panics

Panics if block_size is 0.

pub fn extend( &mut self, tokens: Tokens, ) -> Result<Option<Range<usize>>, TokenBlockError>

Extends the sequence with the given tokens, potentially completing multiple blocks.

This method processes all tokens from the input Tokens object. If adding tokens causes one or more blocks to become full, they are committed and added to the internal list of completed blocks.

Arguments

• tokens - The Tokens object containing the tokens to extend the sequence with.

Returns

• Ok(Some(Range<usize>)) - The range of indices in the blocks vector corresponding to the blocks completed during this extend operation.
• Ok(None) - If no blocks were completed.
• Err(TokenBlockError) - If an internal error occurs during commit.

pub fn append(&mut self, token: Token) -> Result<Option<usize>, TokenBlockError>

Appends a single token to the sequence.

If adding this token completes the current partial block, the block is committed, and the index of the newly completed block is returned.

This method is equivalent to calling [extend] with a single-token Tokens object.

Arguments

• token - The Token to append.

Returns

• Ok(Some(usize)) - The index of the block that was just completed.
• Ok(None) - No block was completed by adding this token.
• Err(TokenBlockError) - If an internal error occurs during processing.

pub fn truncate(&mut self, len: usize) -> Result<(), TokenBlockError>

Shortens the sequence, keeping the first len tokens and removing the rest.

If len is greater than the sequence’s current length, this has no effect.

This operation is analogous to Vec::truncate. It may involve removing tokens from the current partial block, removing entire completed blocks, and adjusting the current partial block to reflect the new end of the sequence.

Arguments

• len - The number of tokens to keep.

Returns

• Ok(()) - If the sequence was successfully truncated.
• Err(TokenBlockError::InsufficientTokens) - This error should ideally not occur if len is correctly checked against total_tokens, but the underlying pop_tokens might return it.

pub fn unwind(&mut self, count: usize) -> Result<(), TokenBlockError>

Removes the last count tokens from the sequence.

This is a convenience method that calculates the required length and calls [truncate].

Arguments

• count - The number of tokens to remove from the end.

Returns

• Ok(()) - If the tokens were successfully removed.
• Err(TokenBlockError::InsufficientTokens) - If count is greater than or equal to the total number of tokens in the sequence.

pub fn reset(&mut self)

Resets the sequence to the initial state.

Clears any accumulated multimodal runs; after reset the sequence behaves identically to a freshly-constructed zero-MM sequence with the same salt_hash and block_size.

pub fn pop(&mut self) -> Option<Token>

Removes the last token from the sequence and returns it, or None if it is empty.

This operation is analogous to Vec::pop.

Returns

• Some(Token) - The last token, if the sequence was not empty.
• None - If the sequence was empty.

Panics

Panics if the sequence has accumulated multimodal runs (see Self::try_pop for a non-panicking variant). pop returns Option<Token> and cannot signal “operation unsupported on MM sequence” through its return type without breaking the Vec::pop analogy and silently lying to callers.

pub fn try_pop(&mut self) -> Result<Option<Token>, TokenBlockError>

Non-panicking variant of Self::pop.

Returns:

• Ok(Some(token)) when the sequence had at least one token and pop succeeded.
• Ok(None) when the sequence was empty.
• Err(TokenBlockError::MmRunsPresent) when the sequence has multimodal runs.

pub fn blocks(&self) -> &[TokenBlock]

Returns a slice containing all the completed TokenBlocks in the sequence.

pub fn last_complete_block(&self) -> Option<&TokenBlock>

Returns a reference to the last completed TokenBlock in the sequence, if any.

pub fn current_block(&self) -> &PartialTokenBlock

Returns a reference to the current PartialTokenBlock where new tokens are added.

pub fn into_parts(self) -> (Vec<TokenBlock>, PartialTokenBlock)

Consumes the sequence and returns its parts: a Vec of completed blocks and the final partial block.

pub fn block_size(&self) -> usize

Returns the block size used for this sequence.

pub fn salt_hash(&self) -> SaltHash

Returns the SaltHash used for this sequence.

pub fn total_tokens(&self) -> usize

Returns the total number of tokens in the sequence (sum of tokens in all completed blocks plus tokens in the current partial block).

pub fn tokens_at(&self, range: Range<usize>) -> Tokens

Extract the token with the range

pub fn split_tokens( tokens: &[Token], block_size: u32, salt_hash: SaltHash, ) -> (Vec<TokenBlock>, PartialTokenBlock)

Splits a Tokens object into a vector of completed blocks and a final partial block.

This is primarily used internally by TokenBlockSequence::new but can be used externally.

Arguments

• tokens - The Tokens to split.
• block_size - The size of each block.
• salt_hash - The SaltHash to use for hashing.

Returns

A tuple containing (Vec<TokenBlock>, PartialTokenBlock).

Panics

Panics if block_size is 0.

pub fn from_slice( tokens: &[Token], block_size: u32, salt_hash: Option<SaltHash>, ) -> Self

Creates a new TokenBlockSequence from a slice of tokens.

The tokens are split into blocks of block_size. Any remaining tokens form the initial current_block.

Arguments

• tokens - The slice of tokens to create the sequence from.
• block_size - The size of each block.
• salt_hash - The SaltHash to use for hashing.

pub fn new_with_mm( tokens: Tokens, mm_info: &[TokenBlockMmInfo], block_size: u32, salt_hash: Option<SaltHash>, ) -> Result<Self, TokenBlockError>

Creates a TokenBlockSequence with multimodal placeholder runs.

mm_info is validated and sorted via validate_and_sort_mm_info. Each block’s BlockHash is computed using the per-block byte encoding documented on compute_block_bytes_with_mm, which selects one of two encodings:

• MM-affected block (at least one run overlaps the block): every slot emits 13 bytes — a 1-byte tag (MM_SLOT_TAG_TOKEN or MM_SLOT_TAG_PLACEHOLDER) followed by a 12-byte payload. Real-token slots carry token_id u32 LE plus 8 bytes of padding; placeholder slots carry run_offset u32 LE | mm_hash u64 LE.
• Non-MM block (no run overlaps): the legacy bytemuck::cast_slice(tokens) form is used (4 bytes per slot, LE u32), preserving cache identity with blocks produced by Self::from_slice.

Returns an error if mm_info is invalid (overlap, out of bounds, zero-length run).

pub fn split_tokens_with_mm( tokens: &[Token], mm_runs: &[TokenBlockMmInfo], block_size: u32, salt_hash: SaltHash, ) -> (Vec<TokenBlock>, PartialTokenBlock)

MM-aware variant of Self::split_tokens.

mm_runs must be pre-validated and sorted (e.g., via validate_and_sort_mm_info).

pub fn mm_runs(&self) -> &[TokenBlockMmInfo]

Returns the validated, sorted multimodal runs accumulated by this sequence.

pub fn push_token( &mut self, token: Token, ) -> Result<Option<usize>, TokenBlockError>

Appends a single real token to the sequence.

Equivalent to Self::append but named for symmetry with Self::push_mm_run.

pub fn push_mm_run( &mut self, mm_hash: u64, length: usize, ) -> Result<Option<Range<usize>>, TokenBlockError>

Appends a multimodal placeholder run of length slots all tagged with mm_hash.

The placeholder run starts at the current end of the sequence (total_tokens() before the call). The token IDs at placeholder slot positions are filled with zero sentinels; hashing uses the (mm_hash, run_offset) pair instead of those token bytes.

Returns the range of fully-committed block indices completed during the call (if any).

pub fn extend_with_mm( &mut self, tokens: &[Token], mm_info: &[TokenBlockMmInfo], ) -> Result<Option<Range<usize>>, TokenBlockError>

Batch-validated extension: appends tokens (with embedded multimodal runs in mm_info) to the sequence in a single validated step.

mm_info offsets are relative to the start of tokens (not the existing sequence). The function validates the chunk, then translates to absolute offsets and applies the updates atomically: it errors before mutating any state if mm_info is invalid.

Real-token regions and placeholder runs are interleaved per the chunk’s layout.

Trait Implementations

impl Debug for TokenBlockSequence

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenBlockSequence

fn eq(&self, other: &TokenBlockSequence) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenBlockSequence