Crate lethe_core_rust 

lethe-core-rust

A high-performance hybrid retrieval engine that combines BM25 lexical search with vector similarity using z-score fusion. Lethe Core provides state-of-the-art context selection for conversational AI and retrieval-augmented generation (RAG) systems.

Features

• Hybrid Retrieval: Combines BM25 lexical search with vector similarity for optimal relevance
• Z-Score Fusion: Normalizes and fuses scores using statistical z-score transformation (α=0.5, β=0.5)
• Hero Configuration: Pre-tuned parameters achieving parity with splade baseline performance
• Gamma Boosting: Context-aware score boosting for code, errors, and technical content
• Chunking Pipeline: Intelligent text segmentation with sentence-level granularity
• Async-First: Built on Tokio for high-performance concurrent operations

Quick Start

use lethe_core_rust::{get_hero_config, apply_zscore_fusion, Candidate};

// Get the hero configuration (optimal for splade parity)
let config = get_hero_config();
println!("Hero config: α={}, β={}", config.alpha, config.beta);
 
// Example candidates from BM25 and vector search
let bm25_candidates = vec![
    Candidate {
        doc_id: "doc1".to_string(),
        score: 0.8,
        text: Some("Rust async programming".to_string()),
        kind: Some("bm25".to_string()),
    },
];
 
let vector_candidates = vec![
    Candidate {
        doc_id: "doc1".to_string(),
        score: 0.9,
        text: Some("Rust async programming".to_string()),
        kind: Some("vector".to_string()),
    },
];
 
// Apply z-score fusion with hero configuration (α=0.5)
let results = apply_zscore_fusion(bm25_candidates, vector_candidates, 0.5);
println!("Fused {} candidates", results.len());

Re-exports

pub use types::*;

pub use error::*;

pub use config::*;

pub use utils::*;

pub use chunker::*;

pub use retrieval::*;

pub use embeddings::*;

pub use hyde::*;

pub use query_understanding::*;

pub use ml_prediction::*;

pub use pipeline::*;

Modules

chunker

config

embeddings

error

hyde

ml_prediction

pipeline

query_understanding

retrieval

types

utils

Functions

apply_zscore_fusion

Process candidates using z-score fusion

get_hero_config

Get a hero configuration for testing and benchmarks

get_hero_config_validated

Get a validated hero configuration against canonical hash