Skip to main content

Crate fetchium_core

Crate fetchium_core 

Source
Expand description

§Fetchium Core Library

AI-native search engine for humans and agents.

fetchium-core contains all algorithms, pipelines, and data structures that power the fetchium CLI and MCP/REST API servers. Everything in this crate is designed for both interactive use (humans) and programmatic use (AI agents).

§Architecture

The core pipeline runs in this order:

Search → Extract (CEP) → Rank (HyperFusion) → Token (QATBE/SCS) →
Validate (6 layers) → Citation → Output (markdown/JSON/segments)
                                   ↑
                 Research (AMRS) ──┘

§Key Algorithms (PRD §8)

AlgorithmDescriptionPRD
CEPContent Extraction Protocol — 5-layer cascade§16
QATBEQuery-Aware Token-Budgeted Extraction — BM25 + semantic§17
SCSSemantic Content Segmentation — 8 segment types§18
PDSProgressive Detail Streaming — 4 tiers§19
HyperFusion8-signal ranking: BM25 + semantic + temporal + authority§21
QADDQuery-Aware DOM Distillation — 5-step DOM pruning§15
AMRSAdaptive Multi-Agent Research Swarm§24
PIEPersistent Intelligence Engine — cross-session learning§32
RARRetry-and-Refine — 5-checkpoint self-correction§44

§Quick Example

use fetchium_core::rank::rerank;
use fetchium_core::types::{BackendId, ResultItem};

// Build some search results
let mut results = vec![
    ResultItem {
        title: "Rust Programming Language".into(),
        url: "https://rust-lang.org".into(),
        snippet: "A language empowering everyone to build reliable software.".into(),
        rank: 0,
        backend: BackendId::DuckDuckGo,
        score: None,
        published_date: None,
    },
    ResultItem {
        title: "Python is great".into(),
        url: "https://python.org".into(),
        snippet: "Python is a versatile scripting language.".into(),
        rank: 1,
        backend: BackendId::DuckDuckGo,
        score: None,
        published_date: None,
    },
];

// Re-rank with BM25 — the Rust result should score higher for "rust"
let ranked = rerank(results, "rust programming language");
assert_eq!(ranked[0].url, "https://rust-lang.org");

§Feature Flags

FeatureDependencyWhat it enables
embeddingsfastembedSemantic scoring, hybrid QATBE, batch embedding
vector-searchusearchHNSW approximate nearest-neighbor index
headlesschromiumoxideCEP layers 3+5, JS rendering, screenshot OCR
mcprmcpMCP protocol support

§Prelude

Import the most commonly used items in one line:

use fetchium_core::prelude::*;

Modules§

ai
AI integration — multi-provider synthesis, model routing, sandwich layout (PRD §23).
api_facade
browser
Browser pool module — managed headless Chromium instances (PRD §8.3).
cache
Cache system — Memory LRU (moka) + SQLite disk (PRD §28).
citation
Citation system (6 styles) + Evidence Graph Protocol (PRD §24).
collab
Collaborative research workspaces — branching, merging, sync (PRD §37).
compare
Comparison research engine — “A vs B vs C” (Phase 5, PRD §10, Mode F).
config
Configuration system for Fetchium (PRD §11).
domain
Domain-specific intelligence modes — academic, code, legal, financial, medical, security (PRD §38).
error
Error types for Fetchium (PRD §44).
evolve
Self-Evolving Architecture — AutoML weight tuning, A/B testing, retrain (PRD §39).
export
Output and export system (Phase 5, PRD §26).
extract
Content Extraction Protocol (CEP) — 5-layer extraction system (PRD §16).
http
HTTP client module — pooled reqwest client with retry logic (PRD §14).
index
Local hybrid search index — Tantivy BM25 + HNSW vector index (Phase 5, PRD §28).
intelligence
Intelligence algorithms — PIE, ToTR, CRP, EDF, SGT, CCE, ACS (PRD §31-39).
monitor
Change detection and URL monitoring engine (Phase 5, PRD §10, Mode G).
multimodal
Multimodal content extraction — video, PDF, OCR, charts (PRD §42).
output
Output formatters — markdown, JSON, CSV, YAML, HTML, segments (PRD §26).
plugin
Plugin system — manifest-based registry + CLI lifecycle (PRD §29).
prelude
Re-export commonly used types.
privacy
Privacy modes — private, tor, air-gap, PII redaction, auto-expire, cache encryption (PRD §36).
proactive
Proactive Intelligence — subscriptions, radar, digest, prefetch, anomaly (PRD §33).
proxy
Proxy rotation system for bypassing search engine IP blocks.
qadd
QADD — Query-Aware DOM Distillation (PRD §8.10).
query
Query understanding and augmentation (Phase 5, PRD §21-22).
rank
Ranking system — BM25, HyperFusion 8-signal, semantic (PRD §21).
research
Research pipeline types and module root (PRD §10).
resilience
Resilience layer — circuit breakers, adaptive rate limiting, health monitoring.
resource
System resource profiling and monitoring (PRD §13).
search
Search backends and orchestrator (PRD §15).
setup
Environment setup utilities — Chromium download, path resolution, dependency checks.
social
Social Media Intelligence System (SMIS).
summarize
AI-powered URL/text summarization pipeline.
telemetry
Pipeline telemetry and observability.
token
Token system — QATBE, SCS, PDS, budget management (PRD SS17-18, SS27).
types
Core data types for Fetchium (PRD §43).
validate
6-layer validation + RAR self-correction (PRD §19, §8.6).
youtube
YouTube Intelligence System (YIS).