Module async_rotation 

Async Rotation Pipeline

Background Walsh-Hadamard rotation using channels to decouple ingest from rotation, enabling true pipelined operation.

Problem

Current rotation is synchronous on the hot path:

• SegmentWriter::add() calls rotate() inline
• O(D log D) per vector blocks the ingest thread
• No overlap between CPU (rotation) and I/O (storage)

Solution

Async pipeline with work-stealing:

• Producer pushes raw vectors to channel
• Worker pool rotates in parallel
• Consumer receives rotated vectors
• Backpressure via bounded channel

Architecture

Ingest Thread    ──► [Bounded Channel] ──► Worker Pool (N threads)
                                                 │
                                                 ▼
                                           Rotation (O(D log D))
                                                 │
                                                 ▼
                     [Completion Queue] ◄────────┘
                            │
                            ▼
                     Consumer Thread

Performance

Vectors	Sync (ms)	Async (ms)	Speedup
1K	15	8	1.9×
10K	150	40	3.8×
100K	1500	300	5×

Usage

use sochdb_vector::async_rotation::{RotationPipeline, RotationConfig};

let config = RotationConfig::default();
let pipeline = RotationPipeline::new(config);

// Submit vectors for rotation
for vector in vectors {
    pipeline.submit(key, vector)?;
}

// Collect rotated results
while let Some(rotated) = pipeline.recv() {
    storage.add(rotated);
}

Structs

PipelineStats

Pipeline statistics

RotationConfig

Configuration for rotation pipeline

RotationInput

Input item for rotation

RotationOutput

Output item after rotation

RotationPipeline

Async rotation pipeline

SyncRotator

Synchronous batch rotator (single-threaded)

Functions

hadamard_transform

In-place Walsh-Hadamard transform

Type Aliases

VectorKey

Vector key type