frankensearch-embed

Embedder implementations for the frankensearch hybrid search library.

Overview

This crate provides three tiers of text embedding, each feature-gated for granular dependency control:

• Hash (hash feature, default): FNV-1a hash embedder with zero ML dependencies. Useful for development, testing, and low-latency scenarios.
• Model2Vec (model2vec feature): potion-128M static embedder (~0.57ms per embed). Serves as the fast tier in two-tier search.
• FastEmbed (fastembed feature): MiniLM-L6-v2 ONNX embedder (~128ms per embed). Serves as the quality tier in two-tier search.

The EmbedderStack auto-detection system probes for locally available models and configures the best fast+quality embedder pair automatically.

Key Types

• EmbedderStack - auto-detected fast+quality embedder pair with optional dimension reduction
• DimReduceEmbedder - wrapper that truncates embeddings to a target dimensionality
• TwoTierAvailability - diagnostic report of which model tiers are available
• HashEmbedder - FNV-1a hash-based embedder (feature: hash)
• Model2VecEmbedder - potion-128M static model embedder (feature: model2vec)
• FastEmbedEmbedder - MiniLM-L6-v2 ONNX embedder (feature: fastembed)
• CachedEmbedder - transparent embedding cache wrapper with hit/miss stats
• BatchCoalescer - batches concurrent embedding requests for throughput optimization
• ModelCacheLayout - manages the on-disk model cache directory structure
• ModelManifest / ModelManifestCatalog - model metadata, lifecycle, and SHA-256 verification
• ModelDownloader - downloads models from HuggingFace with progress tracking (feature: download)
• EmbedderRegistry - registry of known embedder and reranker implementations

Features

Usage

use std::sync::Arc;
use frankensearch_embed::{EmbedderStack, HashEmbedder};
use frankensearch_core::traits::Embedder;

// Simple: use the hash embedder for development
let embedder = HashEmbedder::default_256();

// Production: auto-detect best available models
let stack = EmbedderStack::auto_detect("/path/to/model/cache");
// stack.fast()   -> fastest available embedder
// stack.quality() -> highest quality embedder (if available)

Dependency Graph Position

frankensearch-core
  ^
  |
frankensearch-embed
  ^
  |-- frankensearch-fusion
  |-- frankensearch-fsfs
  |-- frankensearch (root)

License

MIT