Skip to main content

Module chunking

Module chunking 

Source
Expand description

Theta-gamma chunking for wakeup facts (#543, EFF-6).

Working memory holds ~4±1 items as chunks nested in theta cycles, not as a flat list (Lisman/Idiart). The LLM equivalent: semantically related facts grouped under one shared header prime each other and amortize the shared context (category, path prefixes) — fewer tokens, better recall.

Deterministic greedy agglomerative clustering — no RNG, no model: similarity = token-Jaccard over key+value, same-category bonus, and file-path co-reference bonus. Cluster topics are derived lexically (dominant category or shared path token), never via an LLM call.

Structs§

FactCluster

Functions§

cluster_facts
Greedy agglomerative chunking. Input order is the salience order — the first member of each cluster is its most salient fact, and clusters are returned in the order of their founding (= salience) fact.
render_chunked
Render clusters in the compact wakeup notation: one line per chunk, [topic] key=val|key=val. Facts whose category equals the topic drop the category prefix entirely (the header amortizes it) — that is where the token savings over the flat cat/key=val|cat/key=val list come from.