Skip to main content

minify_whitespace

Function minify_whitespace 

Source
pub fn minify_whitespace(source: &str) -> String
Expand description

Reduce indentation waste for embedding by normalizing whitespace.

For each line:

  • Counts leading spaces/tabs, normalises to 2 spaces per indent level (4 spaces → 2, 8 spaces → 4, 1 tab → 2 spaces).
  • Strips trailing whitespace.

Additionally, 3 or more consecutive blank lines are collapsed to a single blank line. This reduces the number of whitespace tokens consumed in the 512-token embedding window without altering visible structure.