Module content 

HTML → markdown/text extraction + token estimation. Owner: parser agent.

Requirements

• extract converts feed item HTML into the requested ContentFormat:
  • Markdown: convert with htmd (HTML → Markdown).
  • Text: render to plain text with html2text.
  • Html: return the HTML as-is (feed-rs already emits sanitized content).
  • None: callers should not call extract; return an empty string defensively.
• On any converter error, fall back to a naive tag strip rather than panicking, and report the fallback so callers can surface a CONTENT_EXTRACTION_FALLBACK warning.
• estimate_tokens returns a cheap, dependency-free token estimate (ceil(chars / 4)), used so agents can budget context.

Constants

TRUNCATION_MARKER

Marker appended to truncated content. The boolean flag is authoritative; this is a human/agent-visible hint that the body was cut.

Functions

content_hash

Stable 16-hex (64-bit) SHA-256 prefix of already-extracted content, for cheap change-detection across runs. Mirrors the item-id construction (lowercase hex, first 8 bytes); 64 bits is ample to flag “this body changed”.

estimate_tokens

Rough token estimate for already-extracted text. Owner: parser agent.

extract

Convert html to the requested format. Owner: parser agent.

truncate_to_chars

Truncate text to at most max_chars characters (not bytes), appending TRUNCATION_MARKER when anything was cut. Returns (text, was_truncated).