Struct IndexWriter 

pub struct IndexWriter<D: DirectoryWriter + 'static> { /* private fields */ }

Async IndexWriter for adding documents and committing segments

Features:

• Parallel indexing with multiple segment builders
• Streams documents to disk immediately (no in-memory document storage)
• Uses string interning for terms (reduced allocations)
• Uses hashbrown HashMap (faster than BTreeMap)

Implementations

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn create( directory: D, schema: Schema, config: IndexConfig, ) -> Result<Self>

Create a new index in the directory

pub async fn create_with_config( directory: D, schema: Schema, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Create a new index with custom builder config

pub async fn open(directory: D, config: IndexConfig) -> Result<Self>

Open an existing index for writing

pub async fn open_with_config( directory: D, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Open an existing index with custom builder config

pub fn schema(&self) -> &Schema

Get the schema

pub fn set_tokenizer<T: Tokenizer>(&mut self, field: Field, tokenizer: T)

Set tokenizer for a field

pub async fn add_document(&self, doc: Document) -> Result<DocId>

Add a document

Documents are distributed randomly across multiple builders for parallel indexing. Random distribution avoids atomic contention and provides better load balancing. When a builder reaches max_docs_per_segment, it is committed and a new one starts.

pub fn pending_build_count(&self) -> usize

Get the number of pending background builds

pub fn pending_merge_count(&self) -> usize

Get the number of pending background merges

pub async fn maybe_merge(&self)

Check merge policy and spawn background merges if needed

This is called automatically after segment builds complete via SegmentManager. Can also be called manually to trigger merge checking.

pub async fn wait_for_merges(&self)

Wait for all pending merges to complete

pub async fn cleanup_orphan_segments(&self) -> Result<usize>

Clean up orphan segment files that are not registered

This can happen if the process halts after segment files are written but before they are registered in segments.json. Call this after opening an index to reclaim disk space from incomplete operations.

Returns the number of orphan segments deleted.

pub async fn get_builder_stats(&self) -> Option<SegmentBuilderStats>

Get current builder statistics for debugging (aggregated from all builders)

pub async fn flush(&self) -> Result<()>

Flush current builders to background processing (non-blocking)

This takes all current builders with documents and spawns background tasks to build them. Returns immediately - use commit() for durability. New documents can continue to be added while segments are being built.

pub async fn commit(&self) -> Result<()>

Commit all pending segments to disk and wait for completion

This flushes any current builders and waits for ALL background builds to complete. Provides durability guarantees - all data is persisted.

pub async fn force_merge(&self) -> Result<()>

Force merge all segments into one