Holographic Memory System (HMS)

Privacy-preserving semantic search using hyperdimensional computing.

A high-performance Holographic Memory System (HMS) for Node.js, powered by Rust. This library implements Vector Symbolic Architectures (VSA) using Binary Spatter Code (BSC) and Sparse Distributed Representations (SDR) to enable semantic search, analogical reasoning, and associative memory at scale.

Developed by WritersLogic -- local-first intelligence with no data leaving your machine.

Features

• Hybrid Retrieval Architecture:
  • NSG (Navigable Small World): Fast proximity graph for approximate nearest neighbors.
  • IVF (Inverted File): Coarse-grained quantization for large datasets.
  • Sparse Inverted Index: Term-based retrieval for high-sparsity queries.
• Symbolic Operations: Native bitwise Binding (XOR), Bundling (Majority Rule), and Permutation (Cyclic Shift).
• Performance Optimized:
  • O(1) Resolution: Cached physical location lookups for instant ID retrieval.
  • FxHash Backend: Ultra-fast non-cryptographic hashing for all retrieval collections.
  • O(N) Selection: Linear-time candidate pruning using select_nth_unstable.
• Persistent Storage: Custom PersistentArena with CRC32 integrity, LZ4 compression, and segmented mmap for crash-safe append-only persistence.
• Meaning Memory: Structured knowledge layer with role-filler algebra, triple stores, multi-hop reasoning, and Hopfield attractor cleanup.
• Cognition Engine: Background discovery of patterns, abstractions, knowledge gaps, hypotheses, and cross-domain analogies from stored triples.
• Graph Engine: Explicit typed relations with multi-hop traversal, transitive/symmetric inference, and temporal filtering.
• Federated Queries: Query across multiple HMS instances in parallel without centralizing data.
• Node.js Bindings: High-efficiency N-API implementation with asynchronous worker thread execution.

Integrations

HMS is available as both a Node.js package and a high-performance Rust crate.

Node.js (N-API)

npm install holographic-memory

Rust (Crates.io)

Add to your Cargo.toml:

[dependencies]
holographic-memory = "0.4"

Core Architecture

HMS is designed for local-first intelligence, combining advanced research in Hyperdimensional Computing with efficient retrieval algorithms.

• Advanced Search: Implements the NSG (Navigable Small World) algorithm, offering high search efficiency and index compactness.
• Adaptive Routing: Employs a retrieval strategy that dynamically switches between graph-based, quantized, and inverted indexing based on dataset statistics.
• Neuro-Symbolic VSA: A robust implementation of Binary Spatter Code (BSC), enabling relational logic $(A \otimes B)$ combined with the associative matching of high-dimensional vector spaces.
• Efficient Data Path: Engineered with a zero-copy N-API interface, $O(1)$ ID resolution, and hardware-aware optimizations for high single-node throughput.

Use Cases

1. Semantic Search & Local RAG

Store text fragments or documents as hypervectors. While HMS uses high-speed Deterministic 3-Gram Encoding for lexical similarity, it also supports LLM Integration. Ingest SOTA embeddings from models like GPT-4 or Llama-3 (via Float32Array) and use HMS as your high-performance retrieval and reasoning layer.

2. Symbolic Knowledge Graphs (Holographic Graph)

Encode relational triplets (Subject, Predicate, Object) into a single hypervector.

• Querying: "What is the capital of France?" becomes (France ⊗ Capital) ⊛ ?.
• Analogies: Solve King : Man :: ? : Woman via holographic vector arithmetic.

3. Real-Time Sequence Pattern Matching

Use Cyclic Permutations to represent order. Ideal for temporal data like time-series patterns, sentence structures, or user behavior trajectories. Querying for a sequence is as fast as querying for a single item.

4. Concept Synthesis & Abstraction

Use the synthesizeConcepts method to identify "abstractions" within your memory. HMS clusters similar hypervectors and generates a centroid that represents the common features of the cluster—essentially "dreaming" up generalized categories from raw data.

5. Explainable Vector Decomposition

Hypervectors in HMS are Distributed Representations. You can use analyzeComponents to decompose a complex bundled vector back into its constituent symbols, providing a "reasoning" trace for why a certain item was retrieved.

Meaning Memory

Meaning Memory is a structured knowledge layer built on top of the holographic vector space. It enables HMS to store, query, and reason over relational knowledge using role-filler algebra rather than flat vector similarity alone.

Architecture

The system is composed of five cooperating modules:

• AtomMemory: Stores individual concept vectors (atoms). Each concept (e.g., "paris", "capital_of") gets a unique high-dimensional vector. Supports deterministic seeding for reproducible embeddings.
• CompositeMemory: Stores composite vectors formed by binding atoms with role assignments. A triple like (paris, capital_of, france) is encoded as a single composite vector using role-shifted XOR binding.
• TripleStore: A symbolic index over (subject, relation, object) triples with four-way FxHash indexing (by subject, relation, object, and composite ID). Enables fast materialized lookups when algebraic decoding is too expensive.
• RoleRegistry: Assigns cyclic-shift values to named roles (subject=0, relation=1, object=3). The asymmetric shifts break XOR commutativity, ensuring (A, r, B) and (B, r, A) produce distinct composites. Custom roles can be registered for n-ary relations.
• IndexedMemory: The shared substrate beneath AtomMemory and CompositeMemory, providing posting-list overlap scanning, IDF weighting with proportional clipping, and tombstone-based soft deletion.

Structural Queries

Structural queries answer relational questions like "What is the capital of France?" by operating on the composite vector space:

1. Compose the known bindings into a partial query vector using role algebra.
2. Overlap scan the composite memory for matching composites (IDF-weighted posting intersection).
3. Admission gating decides the decode path based on fan-out:
   • Algebraic path (fan-out <= threshold): XOR-unbind the query from each composite, inverse-permute by the target role's shift, then run Hopfield attractor cleanup to recover the missing atom.
   • Materialized path (fan-out > threshold): Fall back to TripleStore index lookups for exact symbolic matches.

Multi-Hop Reasoning

Multi-hop queries traverse chains of relations (e.g., "Who is John's grandfather?" via father -> father):

• Rule rewrite: If a CompositionRule maps [father, father] -> grandfather, the query is rewritten as a single algebraic lookup against the derived relation.
• Chained lookup: Otherwise, the system iteratively walks the TripleStore, expanding entities hop-by-hop up to a configurable depth limit.
• Single algebraic: For single-relation queries, the system uses a direct structural query.

Hopfield Attractor Cleanup

After algebraic unbinding, the result vector is noisy. HMS uses a modern Hopfield network with sparse softmax attention to "clean up" the residual vector and snap it to the nearest stored atom:

1. Overlap-scan the noisy vector against AtomMemory's posting lists.
2. Compute sparse softmax attention weights over the top candidates (temperature controlled by beta).
3. Reconstruct a new vector from the attention-weighted sum of stored atoms.
4. Iterate until convergence (similarity > 0.999) or max iterations reached.

Configuration

Enable meaning memory in your config:

let mut config = HmsConfig::default();
config.meaning.enabled = true;
config.meaning.beta = 24.0;              // Hopfield temperature
config.meaning.idf_clip_factor = 3.0;    // IDF clipping (poisoning defense)
config.meaning.algebraic_max_fanout = 40; // Admission control threshold
config.meaning.max_hop_depth = 10;       // Multi-hop chain limit

const hms = new HolographicMemorySystem(16384, './storage', {
  meaning: {
    enabled: true,
    beta: 24.0,
    idfClipFactor: 3.0,
    algebraicMaxFanout: 40,
    maxHopDepth: 10,
  },
});

Cognition Layer

The Cognition layer is a background discovery engine that finds implicit knowledge from stored triples. All components operate with read-only access to meaning memory stores; discovered insights require explicit confirmation before becoming stored facts.

Components

• PatternScanner: Groups triples by relation, counts co-occurring subject/object atoms, and surfaces recurring structural regularities.
• AbstractionEngine: When N entities share the same relation pattern, bundles their atom vectors to create a prototype/category concept.
• GapDetector: Compares an entity's relation profile against its peers to find missing relations (e.g., "most cities have a country, but city X doesn't").
• HypothesisEngine: Proposes fillers for detected gaps by bundling peer data and running Hopfield cleanup to find the nearest stored atom.
• AnalogyDetector: Finds structurally isomorphic domains via connected components and greedy bipartite mapping by relation overlap.
• CognitionLoop: Background thread that runs all components on a configurable interval (default 60s). Insights are collected in a separate buffer without write-locking the main stores.
• MemoryGovernor: Consolidates near-duplicate composites, forgets stale entries, and rebuilds IDF weights. Called explicitly, not automatically.

Configuration

let mut config = HmsConfig::default();
config.meaning.enabled = true;
config.cognition.enabled = true;
config.cognition.interval_secs = 60;
config.cognition.min_pattern_freq = 3;
config.cognition.min_hypothesis_confidence = 0.3;

Core Concepts

Hyperdimensional Computing (HDC)

Traditional AI uses deep vectors (weights). HDC uses high-dimensional (e.g., 10,000+), sparse vectors where information is "holographically" distributed across every dimension.

• Binding (⊗): Combines two vectors into a new, orthogonal vector representing their relationship. Reversible.
• Bundling (⊛): Combines multiple vectors into a single vector that retains similarity to all its components.
• Permutation (Π): Represents sequence and structure by shifting bits.

Quick Start

Semantic Search

const { HolographicMemorySystem } = require('holographic-memory');

async function main() {
  // Initialize with 10,000 dimensions
  const hms = new HolographicMemorySystem(10000, './hms_storage');

  // Memorize associations
  await hms.memorizeText('paris', 'capital of france');
  await hms.memorizeText('berlin', 'capital of germany');

  // Semantic Query
  const results = await hms.query('What is the capital of germany?', 1);
  console.log('Match:', results[0]); // { id: 'berlin', similarity: 0.85 }

  // Analogical Reasoning
  const analogy = await hms.findAnalogy('france', 'paris', 'germany');
  console.log('Result:', analogy[0].id); // 'berlin'
}

main().catch(console.error);

Meaning Memory (Structural Queries)

const { HolographicMemorySystem } = require('holographic-memory');

async function main() {
  const hms = new HolographicMemorySystem(16384, './hms_storage', {
    meaning: { enabled: true },
  });

  // Store relational triples
  await hms.memorizeTriple('paris', 'capital_of', 'france');
  await hms.memorizeTriple('berlin', 'capital_of', 'germany');
  await hms.memorizeTriple('john', 'father', 'mark');
  await hms.memorizeTriple('mark', 'father', 'bob');

  // Structural query: "What is the capital of France?"
  const result = await hms.structuralQuery(
    { subject: 'paris', relation: 'capital_of' },
    'object'
  );
  console.log(result[0].entityId);    // 'france'
  console.log(result[0].confidence);  // 0.98

  // Multi-hop: "Who is John's grandfather?" (father -> father)
  const grandpa = await hms.multiHopQuery('john', ['father', 'father']);
  console.log(grandpa[0].entityId);   // 'bob'
  console.log(grandpa[0].hops.length); // 2
}

main().catch(console.error);

Development

Build Environment

To bypass global permission issues and optimize build performance, use the following configuration:

# Set local cargo home and target directory
export CARGO_HOME=$(pwd)/.cargo_home
export CARGO_TARGET_DIR=/Volumes/C/target

# Build
npm run build

Testing

# Run the 197 unit and integration tests
export CARGO_HOME=$(pwd)/.cargo_home
export CARGO_TARGET_DIR=/Volumes/C/target
cargo test --lib

Ecosystem

Built with HMS

• scrivener-mcp -- MCP server for Scrivener writing projects, using HMS for semantic search across manuscripts and research notes.

Where HMS fits

HMS is a general-purpose vector memory engine. Beyond writing tools, it's well-suited for:

• Local RAG pipelines -- privacy-preserving retrieval for LLM applications without sending data to external APIs
• MCP tool servers -- semantic memory backend for any Model Context Protocol integration
• Knowledge management -- personal knowledge bases with analogical reasoning (Obsidian plugins, Zettlekasten tools)
• Edge/embedded AI -- lightweight enough for single-node deployment; no vector database infrastructure needed
• Research tools -- academic paper similarity, citation graph exploration, concept mapping
• Content moderation -- near-duplicate detection using holographic similarity

Security

HMS is designed with privacy as a core principle. Hyperdimensional vectors are inherently lossy representations; original content cannot be reconstructed from stored vectors.

For security policy and vulnerability reporting, see SECURITY.md.

License

This project is licensed under the Apache License, Version 2.0 - see the LICENSE file for details.