Crate dynamo_tokens

Expand description

Types and utilities for handling sequences of tokens, including block creation and hashing.

Modules§

PartialTokenBlock: Represents a partially filled block of tokens within a sequence.
PositionalLineageHash: A 128-bit positional lineage hash encoding parental lineage for tree traversal.
PositionalRadixTree: Positionally sparse radix tree for efficient indexing of PositionalSequenceHashes.
PositionalSequenceHash: A 128-bit positional sequence hash combining traditional sequence hash with positional information.
TokenBlock: Represents a completed, immutable block of tokens with associated hashes.
TokenBlockMmInfo: Metadata describing a single multimodal placeholder run within a token sequence.
TokenBlockSequence: Represents a sequence of tokens, segmented into fixed-size, hashed blocks.
Tokens: A collection of tokens, represented as a Vec<Token>.

MmInfoError: Errors raised while validating TokenBlockMmInfo inputs.
TokenBlockError: Errors that can occur during PartialTokenBlock operations.

CHAIN_XXH3_SEED: Canonical XXH3 seed used by every chain-step (parent_seq, child_block_hash) → next_seq in this codebase. Must match dynamo_kv_router::protocols::XXH3_SEED — the router’s compute_seq_hash_for_block and the PositionalIndexer’s chain re-computation both route through compute_next_sequence_hash (or equivalently seeded helpers in kv-router), so changing this constant requires all of them to flip in lockstep.
MM_SLOT_TAG_PLACEHOLDER: Slot-tag byte for multimodal placeholder slots. See compute_block_bytes_with_mm.
MM_SLOT_TAG_TOKEN: Slot-tag byte distinguishing real-token slots from multimodal placeholder slots in the per-block byte buffer. See compute_block_bytes_with_mm.

compute_block_bytes_with_mm: Builds the byte buffer used to compute a block’s BlockHash.
compute_block_hash: Canonical BlockHash construction: XXH3 over the per-block byte buffer (already encoded by compute_block_bytes_with_mm or cast_slice for the no-MM path), seeded by SaltHash.
compute_hash_v2: Computes a hash of the data using the given seed (raw u64).
compute_next_sequence_hash: Chain-step for sequence hashing: returns the SequenceHash at position + 1 given the parent’s SequenceHash and the child block’s BlockHash.
compute_salt_hash_from_bytes: Canonical SaltHash construction from a pre-canonicalized salt-payload byte buffer. Application-layer callers should use dynamo_kv_hashing::Request::salt_hash which canonicalizes (salt, lora_name) first; this function is the low-level path.
validate_and_sort_mm_info: Validates mm_info against tokens_len and returns a copy sorted by offset.

BlockHash: A 64-bit hash computed from the tokens within a single block (with optional MM frames), seeded by the request’s SaltHash.
Salt: A salt used for hashing, represented as a vector of bytes. This might encode model architecture, weights, PEFT info, etc.
SaltHash: A 64-bit hash of the salt. Computed once per request and used as the seed for every block-hash computation in that request.
SequenceHash: A 64-bit sequence-aware hash. Equals the BlockHash at position 0 and compute_next_sequence_hash(prev_seq, block_hash) at every subsequent position. Salt propagates through seq_hash[0] since block_hash[0] already encodes it.
Token: A token is represented as a 32-bit unsigned integer.