Struct IndexWriter 

pub struct IndexWriter<D: DirectoryWriter + 'static> { /* private fields */ }

Async IndexWriter for adding documents and committing segments

Features:

• Parallel indexing with multiple segment builders
• Streams documents to disk immediately (no in-memory document storage)
• Uses string interning for terms (reduced allocations)
• Uses hashbrown HashMap (faster than BTreeMap)

State management:

• Building segments: Managed here (pending_builds)
• Committed segments + metadata: Managed by SegmentManager (sole owner of metadata.json)

Implementations

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn build_vector_index(&self) -> Result<()>

Build vector index from accumulated Flat vectors (trains ONCE)

This trains centroids/codebooks from ALL vectors across all segments. Training happens only ONCE - subsequent calls are no-ops if already built.

Note: This is auto-triggered by commit() when threshold is crossed. You typically don’t need to call this manually.

The process:

1. Check if already built (skip if so)
2. Collect all vectors from all segments
3. Train centroids/codebooks based on schema’s index_type
4. Update metadata to mark as built (prevents re-training)

pub async fn total_vector_count(&self) -> usize

Get total vector count across all segments (for threshold checking)

pub async fn is_vector_index_built(&self, field: Field) -> bool

Check if vector index has been built for a field

pub async fn rebuild_vector_index(&self) -> Result<()>

Rebuild vector index by retraining centroids/codebooks

Use this when:

• Significant new data has been added and you want better centroids
• You want to change the number of clusters
• The vector distribution has changed significantly

This resets the Built state to Flat, then triggers a fresh training.

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn create( directory: D, schema: Schema, config: IndexConfig, ) -> Result<Self>

Create a new index in the directory

pub async fn create_with_config( directory: D, schema: Schema, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Create a new index with custom builder config

pub async fn open(directory: D, config: IndexConfig) -> Result<Self>

Open an existing index for writing

pub async fn open_with_config( directory: D, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Open an existing index with custom builder config

pub fn schema(&self) -> &Schema

Get the schema

pub fn set_tokenizer<T: Tokenizer>(&mut self, field: Field, tokenizer: T)

Set tokenizer for a field

pub async fn add_document(&self, doc: Document) -> Result<DocId>

Add a document

Documents are distributed randomly across multiple builders for parallel indexing. Random distribution avoids atomic contention and provides better load balancing. When a builder reaches max_docs_per_segment, it is committed and a new one starts.

pub fn pending_build_count(&self) -> usize

Get the number of pending background builds

pub fn pending_merge_count(&self) -> usize

Get the number of pending background merges

pub async fn maybe_merge(&self)

Check merge policy and spawn background merges if needed

This is called automatically after segment builds complete via SegmentManager. Can also be called manually to trigger merge checking.

pub async fn wait_for_merges(&self)

Wait for all pending merges to complete

pub async fn cleanup_orphan_segments(&self) -> Result<usize>

Clean up orphan segment files that are not registered

This can happen if the process halts after segment files are written but before they are registered in segments.json. Call this after opening an index to reclaim disk space from incomplete operations.

Returns the number of orphan segments deleted.

pub async fn get_builder_stats(&self) -> Option<SegmentBuilderStats>

Get current builder statistics for debugging (aggregated from all builders)

pub async fn flush(&self) -> Result<()>

Flush current builders to background processing (non-blocking)

This takes all current builders with documents and spawns background tasks to build them. Returns immediately - use commit() for durability. New documents can continue to be added while segments are being built.

pub async fn commit(&self) -> Result<()>

Commit all pending segments to disk and wait for completion

This flushes any current builders and waits for ALL background builds and merges to complete. Provides durability guarantees - all data is persisted.

Auto-triggers vector index build when threshold is crossed for any field.

pub async fn force_merge(&self) -> Result<()>

Force merge all segments into one