Skip to main content

Module nc_compress

Module nc_compress 

Source
Expand description

Non-code compression tuning (nc-compress-v1, EPIC 12.14).

The built-in identity/whitespace compressors are conservative — right for code, but they leave easy savings on prose, scraped web text, and Markdown. This module adds two lossless-of-meaning compressors tuned for non-code corpora, registered through the same extension_registry path so they are discoverable and conformance-checked:

  • prose — collapse blank-line runs, strip trailing whitespace, collapse intra-line whitespace, and drop adjacent duplicate lines (common in logs and scraped text).
  • markdown — everything prose does, plus strip HTML comments, drop image /badge syntax, and rewrite [text](url) links to their visible text (removing URL token noise an LLM rarely needs).

Both honor a hard byte budget and are deterministic — the invariants the conformance suite (conformance-v1) enforces on every registered compressor.

Functions§

register_into
Register the non-code compressors into reg. Called from ExtensionRegistry::with_builtins.