Crate topodb_json

Expand description

JSON ↔ engine-type conversions shared by TopoDB’s JSON-speaking front ends (currently topodb-mcp; topodb-cli is next).

Most functions here are pure (no I/O, no Db access) and return Result<_, String> — callers are responsible for mapping the String into their own error type (topodb-mcp’s server.rs maps it to an rmcp::ErrorData: invalid_params for bad input, internal_error otherwise). Nothing here ever panics: an unrepresentable value is always an Err, never an unwrap/expect. Exception: the compose module reads from a Db to plan writes, but never writes itself.

Structs§

EgoParams: Parameters for ego snapshot building.
GraphEdge
GraphNode
GraphSnapshot
GraphTruncation
GraphView
LifecycleCandidate: One decay candidate with its full evidence — everything the judge needs to decide keep|forget without another lookup.
LifecycleParams: Tunables for one sweep. Default is the spec’s policy.
PlannedEntity
ReadScopes: A non-empty set of scopes a read filters by. The non-empty invariant is structural: an empty ScopeSet admits nothing, so an empty read set would make every default read silently return empty.
RememberPlan
RememberRequest
TemporalRewrite: A temporal phrase resolved against the injected reference now_ms.

Enums§

ComposeError: Planning failure: Invalid is a caller-fixable input problem (surface the message verbatim); Engine is a database failure unrelated to the input.

Constants§

ALIAS_EDGE_TYPE
ALIAS_LABEL
ALIAS_NAME_PROP
ARTIFACT_LABEL: Warehouse-derived nodes (crate topodb-warehouse): an Artifact is one raw thing a session touched; a Chunk is a text-indexed window of it.
CHUNK_LABEL
CHUNK_TEXT_PROP
DEFAULT_LOCK_WAIT_MS: Default busy-retry budget (ms) when TOPODB_LOCK_WAIT_MS is unset or unparseable. Shared so the CLI, the MCP stdio server, and the daemon can never drift apart on the value.
DEFAULT_REMEMBER_EDGE_TYPE: Edge type remember uses when the caller doesn’t name one.
ENTITY_LABEL: Label/prop name constants for the two built-in write shapes (create_memory/create-memory, create_entity/create-entity). Single source of truth shared by every front end (topodb-mcp’s default IndexSpec and write tools, topodb-cli’s create-entity/create-memory subcommands) so writes land on exactly the (label, prop) pairs the default spec indexes — search and lookup work out of the box regardless of which front end wrote the data.
ENTITY_NAME_PROP
GRAPH_DEFAULT_LIMIT
GRAPH_MERMAID_INLINE_MAX_NODES
GRAPH_SNAPSHOT_VERSION
GRAPH_TITLE_MAX_CHARS
LIFECYCLE_DEFAULT_LIMIT: How many candidates a sweep reports by default.
LIFECYCLE_HALF_LIFE_DECISION_DAYS: A deliberate tie to the semantic constant: decisions age like standing facts. Change independently if dogfood shows otherwise.
LIFECYCLE_HALF_LIFE_EPISODIC_DAYS: Per-kind staleness half-lives (spec-fixed defaults, tunable per call): an episodic observation goes stale in weeks, a standing fact in months, a how-to in a year.
LIFECYCLE_HALF_LIFE_PROCEDURAL_DAYS
LIFECYCLE_HALF_LIFE_SEMANTIC_DAYS
MEMORY_CONTENT_HASH_PROP: Equality-indexed hash of a memory’s normalized content, used to dedup a re-stored fact to the existing node instead of minting a duplicate. Set by the write front ends (remember/create_memory), never by a caller.
MEMORY_CONTENT_PROP
MEMORY_FORGOTTEN_AT_PROP: Millisecond timestamp at which a memory was forgotten (the forget verb). Distinct from supersession — a forgotten fact was not replaced, just judged not worth keeping. Same tombstone mechanics: recall drops a memory whose value here is <= the query’s now; the node is not deleted.
MEMORY_KINDS: The closed kind vocabulary, in canonical order.
MEMORY_KIND_DECISION
MEMORY_KIND_DEFAULT: What an absent kind prop reads as.
MEMORY_KIND_EPISODIC
MEMORY_KIND_PROCEDURAL
MEMORY_KIND_PROP: Memory taxonomy prop (kind), a Str on Memory nodes: episodic (a dated observation: “CI was red this morning”), semantic (a standing fact: “release tags are per-package”), procedural (a how-to: “publish crates in dependency order”), decision (a resolved choice plus its rationale: “ship as one lean PR”). ABSENT MEANS semantic — no migration; the read side maps a missing prop to the default before filtering. Kind never affects ranking; it exists for the lifecycle decay policy and explicit filtering.
MEMORY_KIND_SEMANTIC
MEMORY_LABEL
MEMORY_SUPERSEDED_AT_PROP: Millisecond timestamp at which a memory was superseded by a newer fact. Set by remember’s supersedes; recall drops a memory whose value here is <= the query’s now (so an as_of before it still sees the old fact). The node is not deleted — supersession dates a fact, keeping its history.
MEMORY_TOMBSTONE_PROPS: The canonical liveness set: a Memory is live iff NONE of these props tombstones it. Every read surface (CLI search, MCP search_memories, dedup, advisories, hygiene) filters on this same set so “live” cannot drift between surfaces.
NEAR_DUP_K: How many near-duplicates to surface at most — enough to notice a redundancy without burying the caller.
NEAR_DUP_REVIEW: Cosine floor below NEAR_DUP_THRESHOLD at which a pair is still surfaced, but only as a weaker "possible" candidate. Measurement showed genuine reworded duplicates can sit as low as ~0.70 — and merely-related facts sit right there too (0.69), so there is NO floor that catches the former without the latter. Set just under that overlap (0.68) so borderline restatements are SURFACED for the caller (an LLM, a native entailment judge) to confirm, rather than silently dropped; the "possible" band is the warning that these need a look, not an automatic merge. Widening recall at the cost of precision is the right trade for an advisory tool where a human/agent makes the final call.
NEAR_DUP_THRESHOLD: Cosine-similarity floor for surfacing a semantic near-duplicate.
SYNONYM_EXPANSION_PROP
SYNONYM_LABEL
SYNONYM_TERM_PROP
TEXT_BAND_MIN_TOKENS: Floor on the SMALLER token set’s size before a text-mode containment score may claim the “likely” band. Whitespace tokens include stopwords, so a short memory’s set is trivially contained in any longer memory that happens to mention the same words — containment 1.0 from 3 shared tokens is weak evidence, not strong. 6 keeps the calibrated canonical pair (“Vega stores its data in postgres”, 6 tokens, 0.833 → likely) exactly at its band.
TEXT_NEAR_DUP_CANDIDATES: BM25 candidate window for the write-time advisory; the scan is exhaustive.
TEXT_NEAR_DUP_CONTAINMENT: Text containment floor for text-based near-duplicate fallback when the embedder isn’t Ready. CONTAINMENT = |∩| / min(|A|,|B|) scores candidates by how completely one fact is contained in another — the canonical restatement shape. Floor 0.7 catches the canonical pair (“Vega stores data in postgres” / “Vega now stores data in sqlite for embedded mode” = 5/6 ≈ 0.833) while leaving unrelated disjoint facts near 0; a short fact fully contained in a longer restatement scores 1.0 exactly (supersession/rewording). See dup_band for confidence levels.
UNSUPPORTED: Error string for both directions of an unrepresentable PropValue: Bytes and DateTime have no JSON counterpart over MCP v0, and any JSON shape that isn’t a string/number/bool (array, object, null) has no PropValue counterpart either.

Functions§

apply_upsert_remap: Correct planned entities against an applier’s upsert remap.
build_ego: Build an ego-view snapshot from seeds and optional query.
build_scope: Build a scope-view snapshot: all entities and memories in the scope, truncated to limit (keeping newest-first by ULID, recording honest dropout counts). All nodes have hop: 0. View is marked “scope” with empty seeds and query.
containment_of_sets: Containment similarity of two precomputed token sets: |∩| / min(|A|,|B|). Returns 1.0 if both sets are empty. Returns 0.0 if exactly one set is empty (no overlap is possible, so containment is a deliberate 0.0 rather than NaN).
content_hash: Stable FNV-1a 64-bit hash of normalized content, hex-encoded. PERSISTED (equality-indexed as content_hash) — the algorithm must never change. Collisions are harmless: dedup always verifies exact normalized content.
default_spec: The ONE canonical default IndexSpec for TopoDB’s built-in write shapes, shared by every front end (topodb-mcp when --spec is omitted; topodb-cli when it creates a brand-new db file). Declares equality on (Entity, name), (Alias, name), and (Synonym, term), and text on (Memory, content), (Entity, name), and (Alias, name), using the shared label and property constants.
dup_band: Confidence band for a near-dup similarity: "likely" at/above the strong floor (NEAR_DUP_THRESHOLD), "possible" in the review band below it.
dup_relation: "supersession" when the pair contradicts (see is_supersession), else "duplicate".
edge_believed_at: Determine whether an edge was “believed” (recorded/present, belief axis) at a given Unix-ms timestamp. Mirrors edge_live_at’s shape exactly, but over the belief-axis fields: inclusive lower bound (recorded_at <= t), exclusive upper bound (superseded_at > t). An edge recorded after t is invisible here regardless of its world-time validity — that’s the whole point of the recorded axis. Edges never superseded (None) are treated as still-believed indefinitely.
edge_live_at: Determine whether an edge is “live” (valid/present) at a given Unix-ms timestamp. The validity window is inclusive on the lower bound (valid_from <= t) and exclusive on the upper bound (valid_to > t). Open edges (no valid_to) are treated as eternally open.
edge_to_json: An EdgeRecord → JSON: id/from/to as ULID strings, type for ty (JSON-friendlier than the Rust keyword-adjacent field name), scope per scope_to_json, props per props_to_json, the world-time bounds valid_from/valid_to (valid_to is null while the edge is open), and the belief-axis bounds recorded_at/superseded_at (superseded_at is null while the edge is open on that axis) — see edge_live_at and edge_believed_at.
entity_dedup_key: In-call dedup key for remember’s entity names: whitespace-collapsed, lowercased — mirroring the engine’s prop-index normalization (prop_index::normalize_str, which is pub(crate) and thus can’t be called from here). Drift between the two only weakens IN-CALL dedup ([“Drew”, “drew”] in one call); cross-call dedup always goes through the engine’s own normalized index via find_existing_entity.
existing_memory: The id of a Memory in write_scope whose normalized content equals content. Hash-bucket lookup, then exact normalized-content verify on every candidate; oldest id wins. Superseded or forgotten memories … are excluded — re-learning a retired fact is a NEW fact, and the tombstone’s as_of history stays intact.
find_existing_entity: lookup is the caller’s collision surface (MCP: default read scopes + write scope + shared; CLI: write scope + shared). Ok(None) means “create it” — covering both no-visible-match and a custom spec without the (Entity, name) equality index (Rejected), which degrades to create-always rather than failing the write.
graph_edge
graph_node
is_supersession: True when the two contents read as a CONTRADICTION rather than a restatement: one asserts a salient token the other negates. Cheap and deterministic — the hint that a high-similarity pair is a supersession (retire the stale one), not a duplicate (merge them). Calibrated in the module tests against a labeled battery.
json_to_f32_vec: A JSON array of finite numbers → Vec<f32>, for raw embeddings (topodb::Op::SetEmbedding) and vector-search queries. Err if v isn’t a JSON array, or any element isn’t a finite number. (The host computes embeddings; TopoDB stores/searches the raw floats.)
json_to_prop_changes: A JSON object of property changes for topodb::Op::SetNodeProps. A null value REMOVES the key (None); any other JSON scalar SETS it (Some(PropValue), via json_to_prop_value). Err if v isn’t a JSON object, or a non-null value isn’t a representable scalar. The null-removes convention is what lets a caller delete a prop over the wire — plain json_to_props has no way to express removal.
json_to_prop_value: serde_json::Value → PropValue. Strings/bools map directly. A JSON integer maps to Int when it fits i64, and is an error when it doesn’t ((i64::MAX, u64::MAX] — silently downgrading it to a lossy Float would corrupt the value); only a genuine non-integer number maps to Float. This is the inverse of prop_value_to_json’s Int/Float handling. Every other JSON shape (array, object, null — and, structurally, anything that would have needed to round-trip through Bytes/DateTime) is UNSUPPORTED.
json_to_props: A JSON object → Props, propagating the first unrepresentable value as Err. Err if v isn’t a JSON object at all.
lifecycle_candidates: The Phase C sweep: rank live memories in scopes by staleness and return the top params.limit with full evidence. Read-only and unbumped end to end. now_ms is injectable for determinism.
lock_wait_budget_ms: Parse a TOPODB_LOCK_WAIT_MS env value into a busy-retry budget. Absent → the default, silently. Present-but-unparseable → the default plus a prefix-less warning (the caller prepends its own program name, since the CLI, the stdio server, and the daemon each identify themselves differently). This is the single source of truth for the parse-and-default policy the three front ends share.
memory_kind_half_life: The kind→half-life map for SEARCH RANKING, built from the same constants the lifecycle decay sweep uses so the two can never drift. Semantic is the map’s default bucket: absent kind reads as semantic everywhere in the system, and non-Memory nodes (entities) deliberately decay on the semantic curve too.
memory_props: Builds a new Memory node’s props: caller extra is validated (the system-maintained keys below are rejected, and content collides via merge_required_prop), then the whitespace-normalized content hash is stamped. The ONE constructor for every front end’s new-memory write (plan_remember, MCP create_memory, CLI create-memory) — so the reserved set cannot drift between surfaces.
merge_required_prop: Builds the Props map for a write tool that has one required, caller-named field (create_memory’s content, create_entity’s name) plus an optional JSON props object of additional metadata. key/value are the required field, already converted to a PropValue; extra is the tool call’s optional props param, converted via json_to_props.
node_superseded
node_title: Entity name, else content preview (≤ GRAPH_TITLE_MAX_CHARS chars, char-boundary safe, ‘…’ suffix when cut), else the label itself.
node_to_json: A NodeRecord → JSON: id/label as strings (ULID via Display for id), scope per scope_to_json, and props per props_to_json. Deliberately omits the embedding field — no MCP v0 tool surfaces vector data (that’s a later concern via dedicated embedding tools).
normalize_content: Normalize memory content for dedup: trim and collapse internal whitespace. Deliberately NOT lowercased — casing can carry meaning in a stored fact.
normalize_edge_type: Canonical form for edge types: Unicode-lowercased, with runs of whitespace, hyphens, and underscores collapsed to a single underscore (leading/trailing separators dropped). "Works At", "works-at", and "works_at" all normalize to "works_at" — one relation, one vocabulary entry, instead of three parallel edge types that silently fragment traversal filters. Every front-end write path (link tool, batch link command, CLI) passes edge types through here; read-side type filters should probe both the raw and normalized forms so edges written before normalization stay reachable. Err on a type that normalizes to empty.
open_with_busy_retry: Calls open until it stops returning Busy or budget_ms elapses. Backoff: 25ms doubling to a 500ms cap, with cheap deterministic-enough jitter (from the clock’s nanoseconds — no rand dependency) so two contending processes don’t retry in lockstep. budget_ms == 0 means exactly one attempt (fail fast). Exhaustion returns Err(Busy).
parse_iso_instant: Resolve an explicit ISO bound to UTC ms. Accepts a date — 2026-08-01, 2026-08, or 2026, resolving to the start of its period — or a UTC datetime YYYY-MM-DDTHH:MM[:SS] with an optional trailing Z, resolving to that exact instant (non-UTC offsets and fractional seconds are rejected: a bound silently shifted by timezone math would be worse than an error). None for anything else. Shared by the MCP created_after/created_before params and the CLI --created-* flags so explicit bounds and the rewriter resolve dates identically; the rewriter itself matches dates only — a datetime inside prose does not trigger a rewrite.
parse_read_scopes: Parses a comma-separated list of shared (case-insensitive) or scope ULIDs. Whitespace around each entry is ignored. Order and duplicates are preserved. Rejects an empty list ("", " , "). Each token is parsed via resolve_scope — the default passed there is never used because every token is required.
parse_temporal_query: Extract one temporal phrase from query, resolved against now_ms (unix ms, UTC). None means the caller searches its original query unmodified — see the module docs for the grammar and boundary rules. Pure and deterministic: same (query, now_ms) → same result; no I/O, no clock reads.
plan_forget: Ops marking ids forgotten (stamp forgotten_at + close open out-edges) plus the forgotten ids. The disconnect motion is plan_supersede’s, but the contract is STRICTER: supersede is a bulk mark that skips already-retired ids; forget is an explicit judgment, so EVERY id must be a live Memory in the write scope and any violation rejects the whole call before ops build. now_ms is a parameter so tests are deterministic.
plan_purge: Phase E: plan the destructive purge. Selects every Memory node in scopes whose ANY tombstone prop holds an Int strictly older than tombstoned_before_ms and returns one Op::RemoveNode per hit plus the ascending-sorted id list (same order). The caller decides whether to submit — the CLI’s dry-run prints this plan without ever writing. Non-Int tombstone values are not marks (the engine’s liveness rule); live nodes are never selected. Purge is deliberately CLI-only and never part of the lifecycle graph: reclamation is an operator action.
plan_remember
plan_supersede: Ops marking ids superseded (stamp + close open out-edges) plus the ids actually marked. Moved from server.rs supersede_ops; now_ms is a parameter so tests are deterministic. Error strings must stay identical. Public building block for layers (e.g. topodb-obsidian) that supersede without going through plan_remember.
prop_value_to_json: PropValue → serde_json::Value. Str/Int/Bool map directly; Float maps to a JSON number (Err only for a non-finite float, which JSON has no representation for). Bytes/DateTime are UNSUPPORTED.
props_to_json: Props (a BTreeMap<String, PropValue>) → a JSON object, propagating the first unrepresentable value as Err.
resolve_batch
resolve_entities_by_name: Canonical entities for name: direct (Entity, name) matches plus (Alias, name) matches followed through alias_of. Deduped by id, oldest first.
resolve_scope: Resolves a tool’s optional scope string param to a Scope: None → default (the server’s configured default scope); Some("shared") (case-insensitive) → Scope::Shared; Some(<ulid>) → Scope::Id; any other string → a clear Err. Mirrors topodb-mcp’s config::parse_scope “shared” / ULID contract, generalized to the Option (tool-call) case.
scope_label: Human/JSON-facing rendering of a Scope: "shared" or the ULID string. Reused by every front end’s info/db_info-style output and scope round-tripping. (Distinct from scope_to_json, which wraps the same rendering in a serde_json::Value for a JSON response body; this returns a bare String for contexts — like a struct field — that want the label without a Value wrapper.)
scope_to_json: A Scope → its JSON rendering: "shared" or the scope’s ULID string. Mirrors the shared/ULID label convention used across TopoDB’s JSON-facing front ends (e.g. topodb-mcp’s db_info tool).
scope_to_scope_set: A resolved Scope → the singleton ScopeSet a read call needs: Shared admits only the shared scope, Id(id) admits only that one scope id.
scopes_to_scope_set: Several resolved Scopes → the ScopeSet a multi-scope read runs against. Scope::Shared sets the set’s include_shared flag; each Scope::Id becomes a member id. This is the only constructor that can produce a genuinely multi-member ScopeSet — scope_to_scope_set always collapses to a singleton, which is why “this project plus shared” was previously unexpressible from any client.
staleness: The spec’s staleness score: (age / half_life) / ln(e + access_count). Negative ages (a node minted after now — clock skew or a backdated sweep) clamp to 0. Pure and exported so tests and future policy layers can pin the formula itself.
subgraph_to_json: A Subgraph → {"nodes": [...], "edges": [...]}, each element per node_to_json/edge_to_json.
text_dup_band: Band for a TEXT-mode (containment) score: dup_band’s cosine-derived cutoffs, except capped at “possible” when the smaller token set is under TEXT_BAND_MIN_TOKENS — the score is real (the pair still surfaces), but small-set containment can’t justify “likely”.
to_canonical_json
to_dot
to_html: Self-contained interactive HTML: the canonical JSON is embedded verbatim (with < escaped as \u003c to make a literal </script> breakout impossible) inside assets/graph.html, a vanilla-JS force-layout viewer.
to_mermaid
tokens: Tokenize a string into a set of lowercase tokens (whitespace-split).
upgraded_spec: Maps a db’s persisted spec forward when — and only when — it is exactly a stock default this crate has shipped: any recognized stock generation upgrades to the current default_spec. Any other spec — a --spec customization, however small — is returned unchanged: silently rewriting a declared spec would reindex data behind its owner’s back. Comparison is order-insensitive, matching how the engine’s ensure_index_spec compares specs (it sorts both lists before persisting).
validate_memory_kind: Enum-validates a caller-supplied memory kind. Exact-match, lowercase only — a taxonomy is a vocabulary, not a suggestion box.