Struct IndexWriter 

pub struct IndexWriter<D: DirectoryWriter + 'static> { /* private fields */ }

Async IndexWriter for adding documents and committing segments.

Backpressure: add_document() is sync, O(1). Returns Error::QueueFull when the shared queue is at capacity — caller must back off.

Two-phase commit:

• prepare_commit() → PreparedCommit::commit() or PreparedCommit::abort()
• commit() is a convenience that does both phases.
• Between prepare and commit, the caller can do external work (WAL, sync, etc.) knowing that abort is possible if something fails.
• Dropping PreparedCommit without calling commit/abort auto-aborts.

Implementations

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn build_vector_index(&self) -> Result<()>

Train vector index from accumulated Flat vectors (manual, not auto-triggered).

1. Acquires a snapshot (segments safe to read)
2. Collects vectors for training
3. Trains centroids/codebooks
4. Updates metadata (marks fields as Built)
5. Publishes to ArcSwap — merges will use these automatically

Existing flat segments get ANN during normal merges. No rebuild needed.

pub async fn rebuild_vector_index(&self) -> Result<()>

Rebuild vector index by retraining centroids/codebooks.

Resets Built state to Flat, clears trained structures, then trains fresh.

impl<D: DirectoryWriter + 'static> IndexWriter<D>

pub async fn create( directory: D, schema: Schema, config: IndexConfig, ) -> Result<Self>

Create a new index in the directory

pub async fn create_with_config( directory: D, schema: Schema, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Create a new index with custom builder config

pub async fn open(directory: D, config: IndexConfig) -> Result<Self>

Open an existing index for writing

pub async fn open_with_config( directory: D, config: IndexConfig, builder_config: SegmentBuilderConfig, ) -> Result<Self>

Open an existing index with custom builder config

pub fn from_index(index: &Index<D>) -> Self

Create an IndexWriter from an existing Index. Shares the SegmentManager for consistent segment lifecycle management.

pub fn schema(&self) -> &Schema

Get the schema

pub fn set_tokenizer<T: Tokenizer>(&mut self, field: Field, tokenizer: T)

Set tokenizer for a field. Propagated to worker threads — takes effect for the next SegmentBuilder they create.

pub async fn init_primary_key_dedup(&mut self) -> Result<()>

Initialize primary key deduplication from committed segments.

Tries to load a cached bloom filter from pk_bloom.bin first. If the cache covers all current segments, the bloom is reused directly (fast path). If new segments appeared since the cache was written, only their keys are iterated (incremental). Falls back to a full rebuild when no cache exists.

The CPU-intensive bloom build is offloaded via spawn_blocking so it does not block the tokio runtime.

No-op if schema has no primary field.

pub fn add_document(&self, doc: Document) -> Result<()>

Add a document to the indexing queue (sync, O(1), lock-free).

Document is moved into the channel (zero-copy). Workers compete to pull it. Returns Error::QueueFull when the queue is at capacity — caller must back off.

pub fn add_documents(&self, documents: Vec<Document>) -> Result<usize>

Add multiple documents to the indexing queue.

Returns the number of documents successfully queued. Stops at the first QueueFull and returns the count queued so far.

pub async fn maybe_merge(&self)

Check merge policy and spawn a background merge if needed.

pub async fn abort_merges(&self)

Abort all in-flight merge tasks without waiting for completion.

pub async fn wait_for_merging_thread(&self)

Wait for the in-flight background merge to complete (if any).

pub async fn wait_for_all_merges(&self)

Wait for all eligible merges to complete, including cascading merges.

pub fn tracker(&self) -> Arc<SegmentTracker>

Get the segment tracker for sharing with readers.

pub async fn acquire_snapshot(&self) -> SegmentSnapshot

Acquire a snapshot of current segments for reading.

pub async fn cleanup_orphan_segments(&self) -> Result<usize>

Clean up orphan segment files not registered in metadata.

pub async fn prepare_commit(&mut self) -> Result<PreparedCommit<'_, D>>

Prepare commit — signal workers to flush, wait for completion, collect segments.

All documents sent via add_document before this call are guaranteed to be written to segment files on disk. Segments are NOT yet registered in metadata — call PreparedCommit::commit() for that.

Workers are NOT destroyed — they flush their builders and wait for resume_workers() to give them a new channel.

add_document will return Closed error until commit/abort resumes workers.

pub async fn commit(&mut self) -> Result<bool>

Commit (convenience): prepare_commit + commit in one call.

Guarantees all prior add_document calls are committed. Vector training is decoupled — call build_vector_index() manually.

pub async fn force_merge(&mut self) -> Result<()>

Force merge all segments into one.

pub async fn reorder(&mut self) -> Result<()>

Reorder all segments via Recursive Graph Bisection (BP) for better BMP pruning.

Each segment is individually rebuilt with record-level BP reordering: ordinals are shuffled across blocks so that similar content clusters tightly.

pub fn segment_manager(&self) -> &Arc<SegmentManager<D>>

Get the segment manager (for background optimizer access).

Trait Implementations

impl<D: DirectoryWriter + 'static> Drop for IndexWriter<D>

fn drop(&mut self)

Executes the destructor for this type.