Struct IndexWriter 

pub struct IndexWriter<D: DirectoryWriter + 'static> { /* private fields */ }

Async IndexWriter for adding documents and committing segments

Features:

• Queue-based parallel indexing with worker tasks
• Streams documents to disk immediately (no in-memory document storage)
• Uses string interning for terms (reduced allocations)
• Uses hashbrown HashMap (faster than BTreeMap)

Architecture:

• add_document() sends to per-worker unbounded channels (non-blocking)
• Round-robin distribution across workers - no mutex contention
• Each worker owns a SegmentBuilder and flushes when memory threshold is reached

State management:

• Building segments: Managed here (pending_builds)
• Committed segments + metadata: Managed by SegmentManager (sole owner of metadata.json)

Implementations

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn build_vector_index(&self) -> Result<()>

Train vector index from accumulated Flat vectors (manual, not auto-triggered).

1. Acquires a snapshot (segments safe to read)
2. Collects vectors for training
3. Trains centroids/codebooks
4. Updates metadata (marks fields as Built)
5. Publishes to ArcSwap — merges will use these automatically

Existing flat segments get ANN during normal merges. No rebuild needed.

pub async fn rebuild_vector_index(&self) -> Result<()>

Rebuild vector index by retraining centroids/codebooks.

Resets Built state to Flat, clears trained structures, then trains fresh.

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn create( directory: D, schema: Schema, config: IndexConfig, ) -> Result<Self>

Create a new index in the directory

pub async fn create_with_config( directory: D, schema: Schema, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Create a new index with custom builder config

pub async fn open(directory: D, config: IndexConfig) -> Result<Self>

Open an existing index for writing

pub async fn open_with_config( directory: D, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Open an existing index with custom builder config

pub fn from_index(index: &Index<D>) -> Self

Create an IndexWriter from an existing Index

This shares the SegmentManager with the Index, ensuring consistent segment lifecycle management.

pub fn schema(&self) -> &Schema

Get the schema

pub fn set_tokenizer<T: Tokenizer>(&mut self, field: Field, tokenizer: T)

Set tokenizer for a field

pub fn add_document(&self, doc: Document) -> Result<DocId>

Add a document to the indexing queue

Documents are sent to per-worker unbounded channels. This is O(1) and never blocks - returns immediately. Workers handle the actual indexing in parallel.

pub fn add_documents(&self, documents: Vec<Document>) -> Result<usize>

Add multiple documents to the indexing queue

Documents are distributed round-robin to workers. Returns immediately - never blocks.

pub fn pending_build_count(&self) -> usize

Get the number of pending background builds

pub async fn maybe_merge(&self)

Check merge policy and spawn background merges if needed

This is called automatically after commit via SegmentManager. Can also be called manually to trigger merge checking.

pub async fn wait_for_merges(&self)

Wait for all pending merges to complete

pub fn tracker(&self) -> Arc<SegmentTracker>

Get the segment tracker for sharing with readers This allows readers to acquire snapshots that prevent segment deletion

pub async fn acquire_snapshot(&self) -> SegmentSnapshot

Acquire a snapshot of current segments for reading The snapshot holds references - segments won’t be deleted while snapshot exists

pub async fn cleanup_orphan_segments(&self) -> Result<usize>

Clean up orphan segment files that are not registered

This can happen if the process halts after segment files are written but before they are registered in segments.json. Call this after opening an index to reclaim disk space from incomplete operations.

Returns the number of orphan segments deleted.

pub async fn flush(&self) -> Result<()>

Flush all workers - serializes in-memory data to segment files on disk

Sends flush signals to all workers, waits for them to acknowledge, then awaits ALL in-flight build JoinHandles. Completed segments are accumulated in flushed_segments but NOT registered in metadata - only commit() does that.

Workers continue running and can accept new documents after flush.

pub async fn commit(&self) -> Result<()>

Commit all pending segments to metadata.

Tantivy-style: flush → atomic commit → merge evaluation. Nothing else. Vector training is decoupled — call build_vector_index() manually.

pub async fn force_merge(&self) -> Result<()>

Force merge all segments into one. Flushes + commits pending docs, then merges. Does NOT trigger background merges — force_merge handles everything itself.