Expand description
§Fetchium Core Library
AI-native search engine for humans and agents.
fetchium-core contains all algorithms, pipelines, and data structures that power the
fetchium CLI and MCP/REST API servers. Everything in this crate is designed for both
interactive use (humans) and programmatic use (AI agents).
§Architecture
The core pipeline runs in this order:
Search → Extract (CEP) → Rank (HyperFusion) → Token (QATBE/SCS) →
Validate (6 layers) → Citation → Output (markdown/JSON/segments)
↑
Research (AMRS) ──┘§Key Algorithms (PRD §8)
| Algorithm | Description | PRD |
|---|---|---|
| CEP | Content Extraction Protocol — 5-layer cascade | §16 |
| QATBE | Query-Aware Token-Budgeted Extraction — BM25 + semantic | §17 |
| SCS | Semantic Content Segmentation — 8 segment types | §18 |
| PDS | Progressive Detail Streaming — 4 tiers | §19 |
| HyperFusion | 8-signal ranking: BM25 + semantic + temporal + authority | §21 |
| QADD | Query-Aware DOM Distillation — 5-step DOM pruning | §15 |
| AMRS | Adaptive Multi-Agent Research Swarm | §24 |
| PIE | Persistent Intelligence Engine — cross-session learning | §32 |
| RAR | Retry-and-Refine — 5-checkpoint self-correction | §44 |
§Quick Example
use fetchium_core::rank::rerank;
use fetchium_core::types::{BackendId, ResultItem};
// Build some search results
let mut results = vec![
ResultItem {
title: "Rust Programming Language".into(),
url: "https://rust-lang.org".into(),
snippet: "A language empowering everyone to build reliable software.".into(),
rank: 0,
backend: BackendId::DuckDuckGo,
score: None,
published_date: None,
},
ResultItem {
title: "Python is great".into(),
url: "https://python.org".into(),
snippet: "Python is a versatile scripting language.".into(),
rank: 1,
backend: BackendId::DuckDuckGo,
score: None,
published_date: None,
},
];
// Re-rank with BM25 — the Rust result should score higher for "rust"
let ranked = rerank(results, "rust programming language");
assert_eq!(ranked[0].url, "https://rust-lang.org");§Feature Flags
| Feature | Dependency | What it enables |
|---|---|---|
embeddings | fastembed | Semantic scoring, hybrid QATBE, batch embedding |
vector-search | usearch | HNSW approximate nearest-neighbor index |
headless | chromiumoxide | CEP layers 3+5, JS rendering, screenshot OCR |
mcp | rmcp | MCP protocol support |
§Prelude
Import the most commonly used items in one line:
use fetchium_core::prelude::*;Modules§
- ai
- AI integration — multi-provider synthesis, model routing, sandwich layout (PRD §23).
- api_
facade - browser
- Browser pool module — managed headless Chromium instances (PRD §8.3).
- cache
- Cache system — Memory LRU (moka) + SQLite disk (PRD §28).
- citation
- Citation system (6 styles) + Evidence Graph Protocol (PRD §24).
- collab
- Collaborative research workspaces — branching, merging, sync (PRD §37).
- compare
- Comparison research engine — “A vs B vs C” (Phase 5, PRD §10, Mode F).
- config
- Configuration system for Fetchium (PRD §11).
- domain
- Domain-specific intelligence modes — academic, code, legal, financial, medical, security (PRD §38).
- error
- Error types for Fetchium (PRD §44).
- evolve
- Self-Evolving Architecture — AutoML weight tuning, A/B testing, retrain (PRD §39).
- export
- Output and export system (Phase 5, PRD §26).
- extract
- Content Extraction Protocol (CEP) — 5-layer extraction system (PRD §16).
- http
- HTTP client module — pooled reqwest client with retry logic (PRD §14).
- index
- Local hybrid search index — Tantivy BM25 + HNSW vector index (Phase 5, PRD §28).
- intelligence
- Intelligence algorithms — PIE, ToTR, CRP, EDF, SGT, CCE, ACS (PRD §31-39).
- monitor
- Change detection and URL monitoring engine (Phase 5, PRD §10, Mode G).
- multimodal
- Multimodal content extraction — video, PDF, OCR, charts (PRD §42).
- output
- Output formatters — markdown, JSON, CSV, YAML, HTML, segments (PRD §26).
- plugin
- Plugin system — manifest-based registry + CLI lifecycle (PRD §29).
- prelude
- Re-export commonly used types.
- privacy
- Privacy modes — private, tor, air-gap, PII redaction, auto-expire, cache encryption (PRD §36).
- proactive
- Proactive Intelligence — subscriptions, radar, digest, prefetch, anomaly (PRD §33).
- proxy
- Proxy rotation system for bypassing search engine IP blocks.
- qadd
- QADD — Query-Aware DOM Distillation (PRD §8.10).
- query
- Query understanding and augmentation (Phase 5, PRD §21-22).
- rank
- Ranking system — BM25, HyperFusion 8-signal, semantic (PRD §21).
- research
- Research pipeline types and module root (PRD §10).
- resilience
- Resilience layer — circuit breakers, adaptive rate limiting, health monitoring.
- resource
- System resource profiling and monitoring (PRD §13).
- search
- Search backends and orchestrator (PRD §15).
- setup
- Environment setup utilities — Chromium download, path resolution, dependency checks.
- social
- Social Media Intelligence System (SMIS).
- summarize
- AI-powered URL/text summarization pipeline.
- telemetry
- Pipeline telemetry and observability.
- token
- Token system — QATBE, SCS, PDS, budget management (PRD SS17-18, SS27).
- types
- Core data types for Fetchium (PRD §43).
- validate
- 6-layer validation + RAR self-correction (PRD §19, §8.6).
- youtube
- YouTube Intelligence System (YIS).