Module tokens 

Types and utilities for handling sequences of tokens, including block creation and hashing.

Modules

blocks

Structs

PartialTokenBlock

Represents a partially filled block of tokens within a sequence.

TokenBlock

Represents a completed, immutable block of tokens with associated hashes.

TokenBlockSequence

Represents a sequence of tokens, segmented into fixed-size, hashed blocks.

Tokens

A collection of tokens, represented as a Vec<Token>.

Enums

TokenBlockError

Errors that can occur during PartialTokenBlock operations.

Functions

compute_hash_v2

Computes a hash of the data using the given seed.

Type Aliases

BlockHash

A 64-bit hash computed only from the tokens within a single block. It uses compute_hash_v2 with the SaltHash as the seed.

Salt

A salt used for hashing, represented as a vector of bytes. This might encode model architecture, weights, PEFT info, etc.

SaltHash

A 64-bit hash of the salt, computed using compute_hash_v2 with a seed of 0. Used as the initial seed for subsequent block hashes.

SequenceHash

A 64-bit sequence-aware hash. It combines the previous block’s SequenceHash (or the SaltHash for the first block) with the current block’s BlockHash using compute_hash_v2 and the SaltHash as the seed.

Token

A token is represented as a 32-bit unsigned integer.