Struct ContentProcessor 

pub struct ContentProcessor { /* private fields */ }

Content processor for HTML documents

Provides methods to clean, extract, and normalize HTML content.

Implementations

impl ContentProcessor

pub fn new(config: ContentProcessorConfig) -> Self

Create a new content processor with the given configuration

pub fn with_defaults() -> Self

Create a content processor with default settings

pub fn with_max_length(max_length: usize) -> Self

Create a content processor with a maximum length limit

pub fn process(&self, raw_html: &str) -> ProcessedContent

Process raw HTML and return cleaned content

This is the main entry point for content processing. It:

1. Removes script, style, and other non-content elements
2. Extracts text from remaining HTML
3. Decodes HTML entities
4. Normalizes whitespace
5. Optionally truncates to max_length

pub fn extract_text(&self, html: &str) -> String

Extract text content from HTML

Uses the scraper crate to parse HTML and extract text nodes, preserving paragraph structure if configured.

pub fn remove_scripts_styles(&self, html: &str) -> String

Remove script, style, and other non-content elements from HTML

This method performs a comprehensive cleanup of HTML by:

• Removing <script> tags and their content
• Removing <style> tags and their content
• Removing <noscript> tags and their content
• Removing HTML comments
• Removing other configured tags

pub fn normalize_whitespace(&self, text: &str) -> String

Normalize whitespace in text

This method:

• Collapses multiple spaces into single spaces
• Normalizes different whitespace characters (tabs, nbsp, etc.)
• Preserves paragraph breaks (double newlines) if structure preservation is enabled
• Trims leading and trailing whitespace

pub fn truncate_with_ellipsis(&self, text: &str, max: usize) -> String

Truncate text with ellipsis at a word boundary

This method truncates text to approximately the given maximum length, breaking at word boundaries to avoid cutting words in half. Appends “…” to indicate truncation.

pub fn decode_html_entities(text: &str) -> String

Decode HTML entities in text

Handles common HTML entities including:

• Named entities (&, <, >, ",  , etc.)
• Numeric entities (', ', etc.)

Trait Implementations

impl Clone for ContentProcessor

fn clone(&self) -> ContentProcessor

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ContentProcessor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.