Struct TokenBlockSequence 

pub struct TokenBlockSequence { /* private fields */ }

Represents a sequence of tokens, segmented into fixed-size, hashed blocks.

This structure manages a series of completed TokenBlocks and one PartialTokenBlock for accumulating incoming tokens. It provides methods for appending tokens (append, extend), removing tokens (pop, truncate, unwind), and accessing sequence information.

Hashing incorporates an initial SaltHash to ensure uniqueness across different contexts (e.g., different models, PEFTs).

Key Hashes:

• BlockHash: Hash of tokens within a single block (seeded by SaltHash).
• SequenceHash: Hash combining the previous block’s SequenceHash and the current block’s BlockHash (also seeded by SaltHash).

Implementations

impl TokenBlockSequence

pub fn new(tokens: Tokens, block_size: u32, salt_hash: Option<SaltHash>) -> Self

Creates a new TokenBlockSequence from an initial set of tokens.

The tokens are split into blocks of block_size. Any remaining tokens form the initial current_block.

Arguments

• tokens - The initial Tokens for the sequence.
• block_size - The fixed size for each TokenBlock. Must be greater than 0.
• salt_hash - An optional SaltHash. Defaults to 0 if None.

Panics

Panics if block_size is 0.

pub fn extend( &mut self, tokens: Tokens, ) -> Result<Option<Range<usize>>, TokenBlockError>

Extends the sequence with the given tokens, potentially completing multiple blocks.

This method processes all tokens from the input Tokens object. If adding tokens causes one or more blocks to become full, they are committed and added to the internal list of completed blocks.

Arguments

• tokens - The Tokens object containing the tokens to extend the sequence with.

Returns

• Ok(Some(Range<usize>)) - The range of indices in the blocks vector corresponding to the blocks completed during this extend operation.
• Ok(None) - If no blocks were completed.
• Err(TokenBlockError) - If an internal error occurs during commit.

pub fn append(&mut self, token: Token) -> Result<Option<usize>, TokenBlockError>

Appends a single token to the sequence.

If adding this token completes the current partial block, the block is committed, and the index of the newly completed block is returned.

This method is equivalent to calling [extend] with a single-token Tokens object.

Arguments

• token - The Token to append.

Returns

• Ok(Some(usize)) - The index of the block that was just completed.
• Ok(None) - No block was completed by adding this token.
• Err(TokenBlockError) - If an internal error occurs during processing.

pub fn truncate(&mut self, len: usize) -> Result<(), TokenBlockError>

Shortens the sequence, keeping the first len tokens and removing the rest.

If len is greater than the sequence’s current length, this has no effect.

This operation is analogous to Vec::truncate. It may involve removing tokens from the current partial block, removing entire completed blocks, and adjusting the current partial block to reflect the new end of the sequence.

Arguments

• len - The number of tokens to keep.

Returns

• Ok(()) - If the sequence was successfully truncated.
• Err(TokenBlockError::InsufficientTokens) - This error should ideally not occur if len is correctly checked against total_tokens, but the underlying pop_tokens might return it.

pub fn unwind(&mut self, count: usize) -> Result<(), TokenBlockError>

Removes the last count tokens from the sequence.

This is a convenience method that calculates the required length and calls [truncate].

Arguments

• count - The number of tokens to remove from the end.

Returns

• Ok(()) - If the tokens were successfully removed.
• Err(TokenBlockError::InsufficientTokens) - If count is greater than or equal to the total number of tokens in the sequence.

pub fn reset(&mut self)

Resets the sequence to the initial state.

pub fn pop(&mut self) -> Option<Token>

Removes the last token from the sequence and returns it, or None if it is empty.

This operation is analogous to Vec::pop.

Returns

• Some(Token) - The last token, if the sequence was not empty.
• None - If the sequence was empty.

pub fn blocks(&self) -> &[TokenBlock]

Returns a slice containing all the completed TokenBlocks in the sequence.

pub fn last_complete_block(&self) -> Option<&TokenBlock>

Returns a reference to the last completed TokenBlock in the sequence, if any.

pub fn current_block(&self) -> &PartialTokenBlock

Returns a reference to the current PartialTokenBlock where new tokens are added.

pub fn into_parts(self) -> (Vec<TokenBlock>, PartialTokenBlock)

Consumes the sequence and returns its parts: a Vec of completed blocks and the final partial block.

pub fn block_size(&self) -> usize

Returns the block size used for this sequence.

pub fn salt_hash(&self) -> SaltHash

Returns the SaltHash used for this sequence.

pub fn total_tokens(&self) -> usize

Returns the total number of tokens in the sequence (sum of tokens in all completed blocks plus tokens in the current partial block).

pub fn tokens_at(&self, range: Range<usize>) -> Tokens

Extract the token with the range

pub fn split_tokens( tokens: &[Token], block_size: u32, salt_hash: u64, ) -> (Vec<TokenBlock>, PartialTokenBlock)

Splits a Tokens object into a vector of completed blocks and a final partial block.

This is primarily used internally by TokenBlockSequence::new but can be used externally.

Arguments

• tokens - The Tokens to split.
• block_size - The size of each block.
• salt_hash - The SaltHash to use for hashing.

Returns

A tuple containing (Vec<TokenBlock>, PartialTokenBlock).

Panics

Panics if block_size is 0.

pub fn from_slice( tokens: &[Token], block_size: u32, salt_hash: Option<SaltHash>, ) -> Self

Trait Implementations

impl Debug for TokenBlockSequence

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenBlockSequence

fn eq(&self, other: &TokenBlockSequence) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenBlockSequence