Expand description
Content Extraction Protocol (CEP) — 5-layer extraction system (PRD §16).
Phase 1: Layers 1-2 (static HTML — CSS selectors + streaming rewriter). Phase 2: Layers 3-5 (headless Chromium, PDF, Screenshot OCR).
Layer selection is automatic and progressive:
- Layer 1: HTML + CSS selectors (scraper) — ~85% of pages
- Layer 2: HTML + streaming rewriter (lol_html) — enhanced boilerplate removal
- Layer 3: Headless JS rendering — SPAs, dynamic content [
headlessfeature] - Layer 4: PDF/document extraction — PDF, DOCX, RTF
- Layer 5: Screenshot OCR — last resort for image-heavy/canvas content [
headlessfeature]
Modules§
- boilerplate
- Boilerplate removal — strip nav, footer, ads, scripts, styles.
- cep_
predictor - CEP layer predictor — pure Rust decision tree (Phase 5, PRD §8.3).
- layer1
- CEP Layer 1: CSS selector extraction using
scrapercrate. - layer2
- CEP Layer 2: Readability-style extraction using
lol_htmlstreaming rewriter. - layer3
- CEP Layer 3 — Headless Chromium JavaScript rendering (PRD §8.3).
- layer4
- CEP Layer 4 — PDF and document extraction (PRD §8.3).
- layer5
- CEP Layer 5 — Screenshot OCR (PRD §8.3).
- pipeline
- CEP extraction pipeline — orchestrates layer selection and escalation (PRD §8.3).
Structs§
- Content
Metadata - Metadata extracted alongside content.
- Extracted
Content - Extracted content from a web page.