Skip to main content

Module condense

Module condense 

Source
Expand description

Condensing a settled page into levels a caller can widen from.

Types and ranking only, no transport: a caller with a snapshot already has everything this needs, and the cache is keyed on settle rather than fetch. Condensing a settled page into levels a caller can widen from.

A settled page is the object this crate has and a byte stream is not: roles, names, bounds and visibility. Handing all of it to a model is the flat context this exists to avoid — crates.io answers semaphore with 654 matches and a semantic tree is larger still.

Five capabilities, ordered by what they cost and by how much they throw away. A caller starts narrow and widens only when the narrow answer was not enough:

capabilityanswersshape
keywordswhat is this page abouta ranked term list
summarywhat does it sayshort sentences, extractive
previewwhat would I seefirst N semantic nodes
structurehow is it laid outthe tree, names and no bodies
contentthe whole thingfull text, the escape hatch

Two rules make the levels usable rather than lossy.

Every level says what it dropped. A summary that does not say it is a summary is indistinguishable from a short page, and a caller cannot tell whether widening would help. Every return carries a Dropped.

Extractive before abstractive. summary is built from sentences the page actually contains, so it can be checked against the page. A generated summary cannot, and a wrong one is worse than none because it reads as authority. Nothing here calls a model; a term-ranking pass over a settled tree is microseconds, a model is milliseconds at best and a network hop at worst, so an algorithm is tried first and measured.

The keyword ranker is only as good as its null. keyword_margin is the acceptance measure: terms from this page against terms from an unrelated page of the same kind, and the margin is the result rather than the raw score. An uncalibrated ranker measures common English, which is what karen-refuter found under Rust tests — a 76.1% null floor beneath a 70% threshold.

Structs§

CondensationCache
Recent condensations, keyed on settle.
CondenseOptions
The knobs a caller sets once. The defaults are the house’s: the first screenful, the first dozen terms, four sentences.
Condensed
One level’s answer, with the account of what it dropped.
CondensedPage
Every level over one settled page, so a cache stores one object.
Dropped
What a level kept, against what was present.
PreviewNode
One node as a preview shows it.
RankedTerm
One ranked term.
SettleKey
What a cache key must include to be about a page that existed.
StructureNode
One node in the structural skeleton.

Enums§

Condensation
One level, as it crosses the wire.
Level
Which level a Condense request wants. Absent means Level::Keywords, the cheapest, so a caller that does not know yet pays the least.

Functions§

condense
Every level at once, so a cache stores one object per settled page.
content
The whole thing: the escape hatch, and the level a caller widens to last.
keyword_margin
The distinctiveness of a page’s terms against a null page of the same kind.
keywords
What the page is about: terms ranked by frequency weighted by semantic role.
preview
What a caller would see first: the leading visible nodes, with their roles.
structure
How the page is laid out: the tree, names kept and text bodies dropped.
summary
What the page says, as sentences the page actually contains.