Struct IndexWriter 

pub struct IndexWriter<D: DirectoryWriter + 'static> { /* private fields */ }

Async IndexWriter for adding documents and committing segments.

Backpressure: add_document() is sync, O(1). Returns Error::QueueFull when the shared queue is at capacity — caller must back off.

Two-phase commit:

• prepare_commit() → PreparedCommit::commit() or PreparedCommit::abort()
• commit() is a convenience that does both phases.
• Between prepare and commit, the caller can do external work (WAL, sync, etc.) knowing that abort is possible if something fails.
• Dropping PreparedCommit without calling commit/abort auto-aborts.

Implementations

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn build_vector_index(&self) -> Result<()>

Train vector index from accumulated Flat vectors (manual, not auto-triggered).

1. Acquires a snapshot (segments safe to read)
2. Collects vectors for training
3. Trains centroids/codebooks
4. Updates metadata (marks fields as Built)
5. Publishes to ArcSwap — merges will use these automatically

Existing flat segments get ANN during normal merges. No rebuild needed.

pub async fn rebuild_vector_index(&self) -> Result<()>

Rebuild vector index by retraining centroids/codebooks.

Resets Built state to Flat, clears trained structures, then trains fresh.

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn create( directory: D, schema: Schema, config: IndexConfig, ) -> Result<Self>

Create a new index in the directory

pub async fn create_with_config( directory: D, schema: Schema, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Create a new index with custom builder config

pub async fn open(directory: D, config: IndexConfig) -> Result<Self>

Open an existing index for writing

pub async fn open_with_config( directory: D, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Open an existing index with custom builder config

pub fn from_index(index: &Index<D>) -> Self

Create an IndexWriter from an existing Index. Shares the SegmentManager for consistent segment lifecycle management.

pub fn schema(&self) -> &Schema

Get the schema

pub fn set_tokenizer<T: Tokenizer>(&mut self, field: Field, tokenizer: T)

Set tokenizer for a field. Propagated to worker threads — takes effect for the next SegmentBuilder they create.

pub fn add_document(&self, doc: Document) -> Result<()>

Add a document to the indexing queue (sync, O(1), lock-free).

Document is moved into the channel (zero-copy). Workers compete to pull it. Returns Error::QueueFull when the queue is at capacity — caller must back off.

pub fn add_documents(&self, documents: Vec<Document>) -> Result<usize>

Add multiple documents to the indexing queue.

Returns the number of documents successfully queued. Stops at the first QueueFull and returns the count queued so far.

pub async fn maybe_merge(&self)

Check merge policy and spawn a background merge if needed.

pub async fn wait_for_merging_thread(&self)

Wait for the in-flight background merge to complete (if any).

pub async fn wait_for_all_merges(&self)

Wait for all eligible merges to complete, including cascading merges.

pub fn tracker(&self) -> Arc<SegmentTracker>

Get the segment tracker for sharing with readers.

pub async fn acquire_snapshot(&self) -> SegmentSnapshot

Acquire a snapshot of current segments for reading.

pub async fn cleanup_orphan_segments(&self) -> Result<usize>

Clean up orphan segment files not registered in metadata.

pub async fn prepare_commit(&mut self) -> Result<PreparedCommit<'_, D>>

Prepare commit — signal workers to flush, wait for completion, collect segments.

All documents sent via add_document before this call are guaranteed to be written to segment files on disk. Segments are NOT yet registered in metadata — call PreparedCommit::commit() for that.

Workers are NOT destroyed — they flush their builders and wait for resume_workers() to give them a new channel.

add_document will return Closed error until commit/abort resumes workers.

pub async fn commit(&mut self) -> Result<()>

Commit (convenience): prepare_commit + commit in one call.

Guarantees all prior add_document calls are committed. Vector training is decoupled — call build_vector_index() manually.

pub async fn force_merge(&mut self) -> Result<()>

Force merge all segments into one.

Trait Implementations

impl<D: DirectoryWriter + 'static> Drop for IndexWriter<D>

fn drop(&mut self)

Executes the destructor for this type.