Expand description
Non-code compression tuning (nc-compress-v1, EPIC 12.14).
The built-in identity/whitespace compressors are conservative — right for
code, but they leave easy savings on prose, scraped web text, and Markdown.
This module adds two lossless-of-meaning compressors tuned for non-code
corpora, registered through the same
extension_registry path so they are
discoverable and conformance-checked:
prose— collapse blank-line runs, strip trailing whitespace, collapse intra-line whitespace, and drop adjacent duplicate lines (common in logs and scraped text).markdown— everythingprosedoes, plus strip HTML comments, drop image /badge syntax, and rewrite[text](url)links to their visible text (removing URL token noise an LLM rarely needs).
Both honor a hard byte budget and are deterministic — the invariants the
conformance suite (conformance-v1) enforces on every registered compressor.
Functions§
- register_
into - Register the non-code compressors into
reg. Called fromExtensionRegistry::with_builtins.