Skip to main content

Module extract

Module extract 

Source
Expand description

Content Extraction Protocol (CEP) — 5-layer extraction system (PRD §16).

Phase 1: Layers 1-2 (static HTML — CSS selectors + streaming rewriter). Phase 2: Layers 3-5 (headless Chromium, PDF, Screenshot OCR).

Layer selection is automatic and progressive:

  • Layer 1: HTML + CSS selectors (scraper) — ~85% of pages
  • Layer 2: HTML + streaming rewriter (lol_html) — enhanced boilerplate removal
  • Layer 3: Headless JS rendering — SPAs, dynamic content [headless feature]
  • Layer 4: PDF/document extraction — PDF, DOCX, RTF
  • Layer 5: Screenshot OCR — last resort for image-heavy/canvas content [headless feature]

Modules§

boilerplate
Boilerplate removal — strip nav, footer, ads, scripts, styles.
cep_predictor
CEP layer predictor — pure Rust decision tree (Phase 5, PRD §8.3).
layer1
CEP Layer 1: CSS selector extraction using scraper crate.
layer2
CEP Layer 2: Readability-style extraction using lol_html streaming rewriter.
layer3
CEP Layer 3 — Headless Chromium JavaScript rendering (PRD §8.3).
layer4
CEP Layer 4 — PDF and document extraction (PRD §8.3).
layer5
CEP Layer 5 — Screenshot OCR (PRD §8.3).
pipeline
CEP extraction pipeline — orchestrates layer selection and escalation (PRD §8.3).

Structs§

ContentMetadata
Metadata extracted alongside content.
ExtractedContent
Extracted content from a web page.