Expand description
Shared lexical vocabulary for deterministic in-kernel value scoring.
Compression utility (goal overlap, directive dependency) and memory curation
(fuzzy dedupe) must speak ONE term vocabulary — and it must discriminate for
CJK text, where there is no whitespace between words: per-character unigrams
make every Chinese unit overlap every Chinese goal, and whitespace tokens
make a whole Chinese sentence a single token. Terms here are lowercased word
runs plus Han character bigrams, matching the host-side recall rankers
(node memory/ranking.ts, python memory/ranking.py) so knowledge
residency, compression, and memory recall share one notion of “relevant”.
Functions§
- jaccard
- Term-set Jaccard similarity over the shared vocabulary.
- overlap_
count - Number of terms shared by both sets.
- terms
- Deterministic term set of
text.