Struct BM25Index 

pub struct BM25Index { /* private fields */ }

BM25 index implementation

Implementations

impl BM25Index

pub fn to_compressed_bytes(&self, compression: Compression) -> Result<Vec<u8>>

Serialize to compressed bytes using specified compression

Errors

Returns error if serialization or compression fails

pub fn from_compressed_bytes( data: &[u8], compression: Compression, ) -> Result<Self>

Deserialize from compressed bytes

Errors

Returns error if decompression or deserialization fails

impl BM25Index

pub fn new() -> Self

Create a new BM25 index with default parameters

pub fn with_params(k1: f32, b: f32) -> Self

Create with custom BM25 parameters

pub fn with_tokenizer(self, tokenizer: Arc<dyn Tokenizer>) -> Self

HELIX-IDEA-005 Phase 4 (FALSIFY-HYBRID-003): plug a custom tokenizer into the index so BM25’s notion of “term” can be shared with other consumers (e.g., an inference path that uses the same lexicon).

When tokenizer is Some, the index’s internal tokenize() delegates to it; the built-in lowercase / stopwords / min-length rules are bypassed entirely. To revert to the built-in path, construct a fresh BM25Index::new().

Arc<dyn Tokenizer> because the index is Clone and may be shared across threads — Box<dyn Tokenizer> would force each clone to deep-copy the tokenizer state.

pub fn has_custom_tokenizer(&self) -> bool

True iff a custom tokenizer is plugged in (used by tests and FALSIFY-HYBRID-003 to confirm the override path is active).

pub fn indexed_terms(&self) -> Vec<&str>

All terms currently indexed (i.e., the keys of the inverted index). Used by FALSIFY-HYBRID-003 to verify the indexer consulted the injected tokenizer during add() — the built-in tokenizer and an injected one produce observably different key sets on the same content.

pub fn with_stopwords(self, stopwords: HashSet<String>) -> Self

Set stopwords

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text. Consults the custom tokenizer override (FALSIFY-HYBRID-003) if one is plugged in, otherwise falls back to the built-in word-boundary + lowercase + stopwords rule.

Trait Implementations

impl Clone for BM25Index

fn clone(&self) -> BM25Index

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BM25Index

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for BM25Index

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for BM25Index

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for BM25Index

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl SparseIndex for BM25Index

fn add(&mut self, chunk: &Chunk)

Index a chunk

fn add_batch(&mut self, chunks: &[Chunk])

Index multiple chunks

fn search(&self, query: &str, k: usize) -> Vec<(ChunkId, f32)>

Search for matching chunks

fn remove(&mut self, chunk_id: ChunkId)

Remove a chunk from the index

fn len(&self) -> usize

Get the number of indexed documents

fn is_empty(&self) -> bool

Check if the index is empty