fetchium_core/lib.rs
1//! # Fetchium Core Library
2//!
3//! AI-native search engine for humans and agents.
4//!
5//! `fetchium-core` contains all algorithms, pipelines, and data structures that power the
6//! `fetchium` CLI and MCP/REST API servers. Everything in this crate is designed for both
7//! interactive use (humans) and programmatic use (AI agents).
8//!
9//! ## Architecture
10//!
11//! The core pipeline runs in this order:
12//!
13//! ```text
14//! Search → Extract (CEP) → Rank (HyperFusion) → Token (QATBE/SCS) →
15//! Validate (6 layers) → Citation → Output (markdown/JSON/segments)
16//! ↑
17//! Research (AMRS) ──┘
18//! ```
19//!
20//! ## Key Algorithms (PRD §8)
21//!
22//! | Algorithm | Description | PRD |
23//! |-----------|-------------|-----|
24//! | **CEP** | Content Extraction Protocol — 5-layer cascade | §16 |
25//! | **QATBE** | Query-Aware Token-Budgeted Extraction — BM25 + semantic | §17 |
26//! | **SCS** | Semantic Content Segmentation — 8 segment types | §18 |
27//! | **PDS** | Progressive Detail Streaming — 4 tiers | §19 |
28//! | **HyperFusion** | 8-signal ranking: BM25 + semantic + temporal + authority | §21 |
29//! | **QADD** | Query-Aware DOM Distillation — 5-step DOM pruning | §15 |
30//! | **AMRS** | Adaptive Multi-Agent Research Swarm | §24 |
31//! | **PIE** | Persistent Intelligence Engine — cross-session learning | §32 |
32//! | **RAR** | Retry-and-Refine — 5-checkpoint self-correction | §44 |
33//!
34//! ## Quick Example
35//!
36//! ```rust
37//! use fetchium_core::rank::rerank;
38//! use fetchium_core::types::{BackendId, ResultItem};
39//!
40//! // Build some search results
41//! let mut results = vec![
42//! ResultItem {
43//! title: "Rust Programming Language".into(),
44//! url: "https://rust-lang.org".into(),
45//! snippet: "A language empowering everyone to build reliable software.".into(),
46//! rank: 0,
47//! backend: BackendId::DuckDuckGo,
48//! score: None,
49//! published_date: None,
50//! },
51//! ResultItem {
52//! title: "Python is great".into(),
53//! url: "https://python.org".into(),
54//! snippet: "Python is a versatile scripting language.".into(),
55//! rank: 1,
56//! backend: BackendId::DuckDuckGo,
57//! score: None,
58//! published_date: None,
59//! },
60//! ];
61//!
62//! // Re-rank with BM25 — the Rust result should score higher for "rust"
63//! let ranked = rerank(results, "rust programming language");
64//! assert_eq!(ranked[0].url, "https://rust-lang.org");
65//! ```
66//!
67//! ## Feature Flags
68//!
69//! | Feature | Dependency | What it enables |
70//! |---------|-----------|-----------------|
71//! | `embeddings` | `fastembed` | Semantic scoring, hybrid QATBE, batch embedding |
72//! | `vector-search` | `usearch` | HNSW approximate nearest-neighbor index |
73//! | `headless` | `chromiumoxide` | CEP layers 3+5, JS rendering, screenshot OCR |
74//! | `mcp` | `rmcp` | MCP protocol support |
75//!
76//! ## Prelude
77//!
78//! Import the most commonly used items in one line:
79//!
80//! ```rust
81//! use fetchium_core::prelude::*;
82//! ```
83
84// Stylistic lints introduced by newer clippy toolchains; the flagged call sites are
85// intentional/borrow-sensitive. Allow crate-wide to keep CI stable across toolchain
86// updates; revisit opportunistically.
87#![allow(clippy::unnecessary_sort_by)]
88#![allow(clippy::manual_checked_ops)]
89
90pub mod api_facade;
91pub mod config;
92pub mod error;
93pub mod types;
94
95// Core pipeline modules
96pub mod cache;
97pub mod extract;
98pub mod http;
99pub mod index;
100pub mod output;
101pub mod rank;
102pub mod resource;
103pub mod search;
104pub mod token;
105
106// Phase 2: QADD and browser pool
107pub mod browser;
108pub mod qadd;
109
110// Phase 3+
111pub mod citation;
112pub mod validate;
113
114// Phase 4+
115pub mod ai;
116pub mod research;
117
118// Phase 5+ (feature-gated optional subsystems)
119#[cfg(feature = "embeddings")]
120pub mod embeddings;
121
122// Always-available Phase 5 modules
123pub mod compare;
124pub mod export;
125pub mod monitor;
126pub mod query;
127
128// Summarization pipeline
129pub mod summarize;
130
131// Phase 6+
132pub mod intelligence;
133
134// Resilience layer — circuit breakers, adaptive rate limiting, bulkhead isolation
135pub mod resilience;
136
137// Telemetry — pipeline observability, metrics, health monitoring
138pub mod telemetry;
139
140// Phase 7+
141pub mod collab;
142pub mod domain;
143pub mod evolve;
144pub mod multimodal;
145pub mod plugin;
146pub mod privacy;
147pub mod proactive;
148pub mod proxy;
149
150// Environment setup utilities (Chromium download, path resolution, checks)
151pub mod setup;
152
153// YouTube Intelligence System
154pub mod youtube;
155
156// Social Media Intelligence System (Twitter/X, Reddit, TikTok, HackerNews + unified)
157pub mod social;
158
159#[cfg(test)]
160pub mod test_utils;
161
162/// Re-export commonly used types.
163pub mod prelude {
164 pub use crate::config::FetchiumConfig;
165 pub use crate::error::{FetchiumError, FetchiumResult};
166 pub use crate::rank::{detect_intent, hyperfusion_rank, QueryIntent};
167 pub use crate::types::*;
168}