Crate nanofts 

NanoFTS - Ultra High-Performance Full-Text Search Engine

A high-performance full-text search engine written in Rust, optimized for billion-scale data with sub-millisecond search response.

Main Features

• LSM-Tree Architecture: No scale limits with efficient disk-based storage
• Incremental Writes: Real-time updates with WAL support
• Fuzzy Search: Similarity-based search with configurable thresholds
• Zero-copy Result Handle: Efficient memory usage with shared bitmap results
• Result Set Operations: AND/OR/NOT operations on search results
• Lazy Load Mode: Memory-efficient loading for large indexes

Quick Start

use nanofts::{UnifiedEngine, EngineConfig};
use std::collections::HashMap;

// Create an in-memory search engine
let engine = UnifiedEngine::new(EngineConfig::memory_only()).unwrap();

// Add a document
let mut fields = HashMap::new();
fields.insert("title".to_string(), "Hello World".to_string());
fields.insert("content".to_string(), "This is a test document".to_string());
engine.add_document(1, fields).unwrap();

// Search
let result = engine.search("hello").unwrap();
println!("Found {} documents", result.total_hits());

// Get document IDs
for doc_id in result.iter() {
    println!("Document ID: {}", doc_id);
}

Persistent Storage

use nanofts::{UnifiedEngine, EngineConfig};

// Create a persistent search engine
let config = EngineConfig::persistent("my_index.nfts")
    .with_lazy_load(true)       // Enable lazy loading for large indexes
    .with_cache_size(10000);    // LRU cache size

let engine = UnifiedEngine::new(config).unwrap();

// ... add documents and search ...

// Flush to disk
engine.flush().unwrap();

Features

• python - Enable Python bindings via PyO3 (disabled by default)
• simd - Enable SIMD acceleration (requires nightly)
• mimalloc - Use mimalloc allocator (enabled by default)

Re-exports

pub use bitmap::FastBitmap;

pub use bitmap::BitmapError;

pub use bitmap::fast_intersection;

pub use bitmap::fast_union;

pub use lsm_single::LsmSingleIndex;

pub use lsm_single::LsmSingleError;

pub use unified_engine::UnifiedEngine;

pub use unified_engine::ResultHandle;

pub use unified_engine::FuzzyConfig;

pub use unified_engine::EngineConfig;

pub use unified_engine::EngineError;

pub use unified_engine::EngineResult;

pub use unified_engine::create_engine;

pub use cache::*;

pub use index::*;

pub use search::*;

pub use shard::*;

Modules

bitmap

High-Performance Roaring Bitmap Operations Module

cache

Lock-Free Concurrent Cache Module

index

Inverted Index Core Module

lsm_single

Single-File LSM Index - Simplified High-Performance Version

search

High-Performance Search Engine Module

shard

Shard Management Module

simd_utils

SIMD Accelerated Utilities Module

unified_engine

Unified Search Engine - Single-File LSM Implementation

vbyte

VByte Variable-Length Encoding Module

wal

Write-Ahead Log (WAL) Module