Expand description
Condensing a settled page into levels a caller can widen from.
Types and ranking only, no transport: a caller with a snapshot already has everything this needs, and the cache is keyed on settle rather than fetch. Condensing a settled page into levels a caller can widen from.
A settled page is the object this crate has and a byte stream is not: roles,
names, bounds and visibility. Handing all of it to a model is the flat
context this exists to avoid — crates.io answers semaphore with 654
matches and a semantic tree is larger still.
Five capabilities, ordered by what they cost and by how much they throw away. A caller starts narrow and widens only when the narrow answer was not enough:
| capability | answers | shape |
|---|---|---|
keywords | what is this page about | a ranked term list |
summary | what does it say | short sentences, extractive |
preview | what would I see | first N semantic nodes |
structure | how is it laid out | the tree, names and no bodies |
content | the whole thing | full text, the escape hatch |
Two rules make the levels usable rather than lossy.
Every level says what it dropped. A summary that does not say it is a
summary is indistinguishable from a short page, and a caller cannot tell
whether widening would help. Every return carries a Dropped.
Extractive before abstractive. summary is built from sentences the
page actually contains, so it can be checked against the page. A generated
summary cannot, and a wrong one is worse than none because it reads as
authority. Nothing here calls a model; a term-ranking pass over a settled
tree is microseconds, a model is milliseconds at best and a network hop at
worst, so an algorithm is tried first and measured.
The keyword ranker is only as good as its null. keyword_margin is the
acceptance measure: terms from this page against terms from an unrelated
page of the same kind, and the margin is the result rather than the raw
score. An uncalibrated ranker measures common English, which is what
karen-refuter found under Rust tests — a 76.1% null floor beneath a 70%
threshold.
Structs§
- Condensation
Cache - Recent condensations, keyed on settle.
- Condense
Options - The knobs a caller sets once. The defaults are the house’s: the first screenful, the first dozen terms, four sentences.
- Condensed
- One level’s answer, with the account of what it dropped.
- Condensed
Page - Every level over one settled page, so a cache stores one object.
- Dropped
- What a level kept, against what was present.
- Preview
Node - One node as a preview shows it.
- Ranked
Term - One ranked term.
- Settle
Key - What a cache key must include to be about a page that existed.
- Structure
Node - One node in the structural skeleton.
Enums§
- Condensation
- One level, as it crosses the wire.
- Level
- Which level a
Condenserequest wants. Absent meansLevel::Keywords, the cheapest, so a caller that does not know yet pays the least.
Functions§
- condense
- Every level at once, so a cache stores one object per settled page.
- content
- The whole thing: the escape hatch, and the level a caller widens to last.
- keyword_
margin - The distinctiveness of a page’s terms against a null page of the same kind.
- keywords
- What the page is about: terms ranked by frequency weighted by semantic role.
- preview
- What a caller would see first: the leading visible nodes, with their roles.
- structure
- How the page is laid out: the tree, names kept and text bodies dropped.
- summary
- What the page says, as sentences the page actually contains.