Module distributed_cache 

Distributed KV cache coordination for multi-GPU inference.

When running pipeline- or tensor-parallel inference, the KV cache for each attention layer must be kept in sync across devices. This module provides coordination types and protocols — not the actual cache storage (which lives in fuel_inference).

Concepts

• Cache shard: One device’s portion of the KV cache, identified by rank.
• Sync event: A broadcast notification that a shard has been updated.
• Distributed prefix: For prefix caching, all ranks must agree on which prefix tokens are cached before reusing entries.

Example

use fuel_parallel::distributed_cache::{
    CacheShardInfo, CacheSyncProtocol, SyncEvent,
};

// Build shard info for rank 0 of 4
let shard = CacheShardInfo::new(0, 4, 32); // rank=0, world=4, num_layers=32
assert_eq!(shard.layers().len(), 8); // 32 layers / 4 ranks = 8 per shard
assert_eq!(shard.layers(), &[0, 1, 2, 3, 4, 5, 6, 7]);

// Protocol tracks sync state
let mut protocol = CacheSyncProtocol::new(4, 32);
protocol.record_event(SyncEvent::ShardUpdated { rank: 0, layer: 3, seq_pos: 42 });
assert_eq!(protocol.latest_position(0, 3), Some(42));

Structs

CacheRoutingHint

Describes which ranks need cache data for a given request.

CacheShardInfo

Identifies a shard of the distributed cache.

CacheSyncProtocol

Tracks distributed cache state across all ranks.

Enums

SyncEvent

Cache synchronisation events exchanged between ranks.