Module knowledge

Expand description

Format v2 “knowledge without forgetting” (spec §9.2–§9.5): a FROZEN genome, skill records v2 that own their tensors over it, and the backbone-gated router policy.

Three required_features bits carry the semantics, each DERIVED by the writers from header content and checked both ways on open:

GENOME ⇔ header.genome — the reader recomputes trunk_hash from the directory and refuses a file whose trunk changed without a new genome;
SKILLS_V2 ⇔ some skills[i].kind — every v2 record is validated against the genome it is bound to and against the directory;
ROUTER_V2 ⇔ header.router — only the declared backbone_gated policy may auto-route; the legacy argmin-of-skills is not a fallback.

Everything here is pure metadata logic (no I/O): format.rs calls validate_knowledge from open(), from every writer and from the tail append, so a file the writer produced is a file the reader accepts.

Modules§

expert_leaf: Per-expert tensor leaves of an MoE layer (model.layers.{l}.mlp.experts.{e}.{leaf}), in the order a record lists them.
lookup_leaf: Tensor leaves of a lookup record (skill.{id}. + leaf).
lookup_policy: Values of LookupInfo.policy (spec §9.5.2).
skill_kind: Skill-record kinds (SkillRecord.kind).
state_switch: StateEffect.switch values.

Structs§

ExecHashes: hash64 of the execution-relevant sections outside the directory: the DTG-MA mask catalog and the sparse index derived from it (None = the section is absent).
ExpertAppend: Parameters of a kind = "expert_append" record (SkillRecord.experts): count new experts in EACH of SkillRecord.layers (the grown MoE layers, any non-empty subset), appended after the experts already present in that layer. The per-layer index of the new experts is not stored — the reader computes it (expert_append_base).
ExpertTensorSpec: One tensor an expert_append record carries.
GenomeInfo: The frozen genome the file carries (header.genome, bit GENOME).
GenomeParent
LineageEvent: One journal entry (header.lineage): birth | skill_committed | skill_retired | requant | recalibrate | compact | rollback. seq is strictly increasing.
LookupInfo: Parameters of a kind = "lookup" record (SkillRecord.lookup, spec §9.5.2): entries cards, each with one text slot per language of langs (slot entry · L + lang), reached through keys normalised, hashed keys. The table itself is the four tensors of lookup_leaf; a lookup record has no overrides, no experts, no layers and the transparent lookup_state_effect — it never touches the network.
PhiSpec: Canonical φ(q) of a user message q: ids = prefix_ids ++ encode(q) ++ suffix_ids (encode without BOS or special handling), run the backbone through layer, take the hidden AFTER that layer, mean over the positions of encode(q) only (span_mean), unit-normalize (unit).
RouterPolicy: header.router (bit ROUTER_V2): the per-request backbone-gated policy. The backbone is the default; a skill runs only when the calibrated decision says so (cortiq-engine router::decide_backbone_gated).
Segment: Data-range ownership after tail appends (header.segments). Offsets are relative to data_off, like directory offsets. A full rewrite is a compaction and clears the table.
SkillBound: The genome a v2 skill is bound to.
SkillOverride: A base tensor a v2 skill replaces.
StateEffect: What switching this skill does to the O(1) sequence state.

Constants§

BACKBONE_CLASS_ID: Row id of the backbone class in router-v2 decisions. No skill may use it.
GENOME_STATUSES: genome.status values.
KEY_NORM: The key normalisation a lookup record declares (LookupInfo.key_norm): KEY_NORM_V2, normalize_key. This reader knows exactly one.
KEY_NORM_V1: The pre-release key normalisation (NFC → lowercase → every non-alphanumeric char → space): it split a word at a combining stress mark or a soft hyphen (Рома́шка → рома шка) and kept Latin diacritics (Pínus ≠ Pinus). No table was released under it; this reader refuses the id with a rebuild hint.
KEY_NORM_V2: cmf-key-v2: normalize_key — the rule every table is built and read with. A new rule is a new id, never a silent change.
KNOWLEDGE_BITS: Knowledge bits (all three are derived from content).
LOOKUP_MAX_NGRAM: Longest key the runtime can match, in words of the normalised form: the user message is searched as n-grams of 1..=LOOKUP_MAX_NGRAM words (cortiq_engine::lookup::MAX_NGRAM is this value), so the builder drops a longer key as unreachable instead of storing it.
METRIC_MSE_UNIT: The metric every router-v2 descriptor uses (unit-normalized φ).
RESERVED_SKILL_IDS: Ids a v2 skill may not take: the CLI and the route reports spell the backbone and the automatic decision with them (--skill none, --route backbone|auto, target: "backbone"), so a skill with such an id would be unreachable or indistinguishable from the backbone.
SEGMENT_KINDS: segments[i].kind values.
SKILL_STATUSES: skills[i].status values (v2 records).

Functions§

canonical_json: Canonical JSON bytes of a value: object keys sorted bytewise at every depth, no whitespace. Independent of map insertion order and of serde_json’s preserve_order feature (which the engine enables and a core-only build does not).
check_lookup_policies: The writers’ rule: every lookup record’s lookup.policy, when present, is one this writer knows — append_skill, update_header_append and the full writers refuse anything else before a byte is written (open() only warns, unknown_lookup_policies).
expert_append_base: First expert index the expert_append record at position at of header.skills owns in layer — the per-layer chain rule: E0 + Σ count of the expert_append records BEFORE at that grow layer. Its experts are base + k, k in 0..count; a layer the record does not grow keeps its experts. at == header.skills.len() plans a record about to be appended.
expert_append_layout: Every tensor (name and shape) the expert_append record record must carry when it sits at position at of header.skills — exactly these, no others: per grown layer l and k in 0..count, expert e = expert_append_base(l) + k: gate_proj.weight, up_proj.weight, down_proj.weight shaped like the trunk’s expert 0 of l, desc.mu [hidden], desc.u shaped like the trunk’s expert 0 desc.u (absent when rank == 0), desc.bias [1] and desc.shell [1] (both f32).
expert_append_state_effect: The state effect of an expert_append record: the grown experts are part of the organism from the first grown layer on and are never switched, so sequence_start (the least permissive) and no state added.
expert_append_tensor_name: skill.{id}.model.layers.{layer}.mlp.experts.{expert}.{leaf}.
expert_tensor_name: model.layers.{layer}.mlp.experts.{expert}.{leaf} — a trunk expert tensor; a record’s copy is skill.{id}. + this (expert_append_tensor_name).
f32_le_head: First f32 (LE) of a payload.
ffn_replace_state_effect: The state effect of replacing the FFN of layers, computed by the WRITER from arch.layer_types: the FFN output of layer l feeds every layer j > l, so switching is transparent iff no layer after min(layers) carries sequence state (and the stack is not looped — a loop feeds layer l back into layer 0). The MTP head is not counted: its state only steers drafts that the main model verifies.
genome_moe_experts: E0 of a genome: genome.moe_experts, or — on a file written before the field existed — arch.moe.num_experts of a resonance-routed MoE (the value the writer would fill). Errors: no genome, or the genome is not a resonance-routed MoE (a gated MoE cannot take appended experts: its gate tensor fixes E).
hex64: {:016x} of a hash64.
is_trunk_tensor: Is this directory entry part of the trunk (not a skill or router tensor)?
key_hash: hash64 of normalize_key(key) — the value stored in keys.hash.
knowledge_bits: The knowledge bits this header’s content requires.
layer_type_is_stateful: Does this layer type carry sequence state (KV, ring, recurrent)? Exhaustive on purpose: a new operator must decide here.
lookup_state_effect: The state effect of a lookup record: it never touches the network, so first_affected_layer 0, transparent, no state added — and the validator requires exactly this.
lookup_tensor_name: skill.{id}.{leaf} for one of lookup_leaf.
lookup_tensors: The four tensors of the lookup record id from hashed keys and slot texts: keys = (key_hash, entry) in any order, slots[entry · L + lang] = the JSON object text of that card in that language (E · L of them, L = info.langs.len()). Sorts the keys by hash and refuses a duplicate hash (two keys that normalise alike — the builder drops and reports them BEFORE calling this), an entry ≥ E, a key count or slot count that disagrees with info, a slot that is not a JSON object. What comes back passes the record validation by construction.
moe_layers: The layers whose trunk carries expert tensors (model.layers.{l}.mlp.experts.0.gate_proj.weight), ascending — the layers an expert_append record may grow (the trainer’s default: all).
normalize_key: cmf-key-v2 (KEY_NORM): NFC → every combining mark and every invisible format character is dropped WITHOUT ending the word (a stress mark over a Cyrillic vowel has no precomposed form, so NFC leaves U+0301 standing: Рома́шка → ромашка; soft hyphen, zero-width space / joiner / non-joiner, word joiner, BOM: ромашка → ромашка) → a precomposed LATIN letter loses its diacritics (Pínus → pinus, école → ecole; a Cyrillic letter keeps its mark — й, ё, ў are letters, not accents) → lowercase → every other char that is not alphanumeric (Unicode) becomes a space → runs of spaces collapse → trim. Shared by the builder (keys) and the runtime (n-grams of the user message), so normalize_key("Пихта бальзамическая (Abies balsamea)") == "пихта бальзамическая abies balsamea". Idempotent.
normalized_key_hash: hash64 of the UTF-8 bytes of an ALREADY normalised key (the runtime normalises the whole message once and hashes its n-grams).
parse_hex64: Parse a hex hash64 (as written by hex64).
read_u32_le: Little-endian u32 words of a raw payload (u32 tensors).
read_u64_le: Little-endian u64 words of a raw payload (u64 tensors).
trunk_expert_rank: Rank of the trunk’s expert 0 descriptor subspace in layer (experts.0.desc.u elements / hidden_size), 0 when the trunk has no desc.u there. Errors: layer is not an MoE layer of the trunk, or it has no per-expert descriptors (experts.0.desc.mu) — the legacy stacked mlp.desc.* form cannot be grown per expert.
trunk_hash: hash64 of the genome’s trunk. Input, in order:
trunk_hash_exec: trunk_hash with the mask / sparse-index sections (NF-1): when the file carries either, 0x1e b"exec" 0x00 + per section (tag u8 + hash64 LE) is appended to the hash input. A file with neither hashes exactly as trunk_hash (so genomes born without masks keep their hash), and a mask catalog added to a genome later changes the trunk hash — the reader refuses it unless it is a new genome.
trunk_hash_json: trunk_hash over the raw arch / tokenizer_config JSON values of a header (Value::Null for an absent bundle) — the reader’s form.
trunk_hash_json_exec: trunk_hash_json with the mask / sparse-index sections (trunk_hash_exec).
unknown_lookup_policies: (skill id, value) of every lookup record whose lookup.policy this reader does not know (lookup_policy::ALL). open() accepts such a file and warns; the runtime reads the record as lookup_policy::DEFAULT — the conservative reading, the one a reader from before the field existed applies (review KF-6).
utc_now_rfc3339: Current UTC time as RFC 3339 (YYYY-MM-DDTHH:MM:SSZ), no dependency.
validate_expert_append_values: The value rules of expert_append descriptors, which the directory cannot express: every desc.bias is finite (the trainer freezes it — 0.0 or the source’s balancing bias, origin.bias_mode; NaN/inf would buy or lose every token) and every desc.shell is finite (+inf would mean “no shell”, NaN would never fire). f32_of yields the first f32 of an entry’s payload (None = not available → refused). Called by open() (mmap) and by every writer (the payload it writes) after validate_knowledge, which fixed the dtype to f32.
validate_knowledge: Every knowledge rule (§9.2–§9.5) over a header + directory: the bits ⇔ content in both directions, the genome’s trunk hash, the lineage order, segment bounds, every v2 skill record and the router policy. Called by open() (with the envelope’s bits) and by every writer before a byte is written. Files carrying none of the new bits and no v2 content pass untouched.
validate_lookup_values: The value rules of lookup tables, which the directory cannot express: keys.hash strictly ascending (sorted, unique — the runtime binary-searches it), every keys.entry < E, entries.off monotone and within the blob, the blob valid UTF-8 with every slot boundary on a character boundary, every slot shaped as a JSON object (first non-blank byte {, last }). With deep every slot is also parsed as JSON: what a WRITER does with the payload it is about to commit; open() passes false — the runtime reads a slot lazily and reports a slot that does not parse at that request, so parsing every card of a table of hundreds of MB on every open of the file would only cost time. bytes_of yields an entry’s payload (None = not available → refused). Called after validate_knowledge, which fixed dtypes and shapes. One pass over the table: O(K + N).