Struct EmbrFS 

pub struct EmbrFS {
    pub manifest: Manifest,
    pub engram: Engram,
    pub resonator: Option<Resonator>,
    /* private fields */
}

EmbrFS - Holographic Filesystem with Guaranteed Reconstruction

100% Reconstruction Guarantee

EmbrFS guarantees bit-perfect file reconstruction through a layered approach:

1. Encode: Data chunks → SparseVec via reversible encoding
2. Verify: Immediately decode and compare to original
3. Correct: Store minimal correction if any difference exists
4. Extract: Decode + apply correction = exact original bytes

This guarantee holds regardless of:

• Data content (binary, text, compressed, encrypted)
• File size (single byte to gigabytes)
• Number of files in the engram
• Superposition crosstalk in bundles

Holographic Mode

When created with new_holographic(), uses ReversibleVSAEncoder which achieves ~94% uncorrected accuracy through position-aware VSA binding. This results in <10% correction overhead instead of the ~200%+ overhead of legacy encoding.

Examples

use embeddenator_fs::EmbrFS;
use std::path::Path;

// Legacy mode (not recommended)
let mut fs_legacy = EmbrFS::new();

// Holographic mode (recommended - minimal storage overhead)
let mut fs = EmbrFS::new_holographic();
assert_eq!(fs.manifest.total_chunks, 0);
assert_eq!(fs.manifest.files.len(), 0);

Fields

manifest: Manifest

engram: Engram

resonator: Option<Resonator>

Implementations

impl EmbrFS

pub fn new() -> Self

👎Deprecated since 0.25.0: Use new_holographic() instead for ~94% encoding accuracy and <10% storage overhead

Create a new empty EmbrFS instance (legacy mode - NOT RECOMMENDED)

This constructor creates an EmbrFS with legacy encoding that has only ~10% accuracy, resulting in ~200%+ storage overhead due to verbatim corrections.

Use new_holographic() instead for production use.

Examples

use embeddenator_fs::EmbrFS;

// Legacy mode - high storage overhead
let fs = EmbrFS::new();
assert_eq!(fs.manifest.files.len(), 0);

pub fn new_holographic() -> Self

Create a new EmbrFS with holographic encoding (RECOMMENDED)

Uses ReversibleVSAEncoder which achieves ~94% uncorrected accuracy through position-aware VSA binding. This results in <10% correction overhead instead of the ~200%+ overhead of legacy encoding.

Examples

use embeddenator_fs::EmbrFS;

let fs = EmbrFS::new_holographic();
assert_eq!(fs.manifest.files.len(), 0);
assert_eq!(fs.manifest.total_chunks, 0);
assert!(fs.is_holographic());

pub fn is_holographic(&self) -> bool

Check if holographic mode is enabled

pub fn chunk_size(&self) -> usize

Get the chunk size being used

pub fn set_resonator(&mut self, resonator: Resonator)

Set the resonator for enhanced pattern recovery during extraction

Configures a resonator network that can perform pattern completion to recover missing or corrupted data chunks during filesystem extraction. The resonator acts as a content-addressable memory that can reconstruct lost information by finding the best matching patterns in its trained codebook.

How it works

• The resonator maintains a codebook of known vector patterns
• During extraction, missing chunks are projected onto the closest known pattern
• This enables robust recovery from partial data loss or corruption

Why this matters

• Provides fault tolerance for holographic storage systems
• Enables reconstruction even when some chunks are unavailable
• Supports graceful degradation rather than complete failure

Arguments

• resonator - A trained resonator network for pattern completion

Examples

use embeddenator_fs::{EmbrFS, Resonator};

let mut fs = EmbrFS::new();
let resonator = Resonator::new();
fs.set_resonator(resonator);
// Now extraction will use resonator-enhanced recovery

pub fn correction_stats(&self) -> CorrectionStats

Get correction statistics for this engram

Returns statistics about how many chunks needed correction and the overhead incurred by storing corrections.

Examples

use embeddenator_fs::EmbrFS;

let fs = EmbrFS::new();
let stats = fs.correction_stats();
assert_eq!(stats.total_chunks, 0);

pub fn ingest_directory<P: AsRef<Path>>( &mut self, dir: P, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Ingest an entire directory into engram format

pub fn ingest_directory_with_prefix<P: AsRef<Path>>( &mut self, dir: P, logical_prefix: Option<&str>, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Ingest a directory into the engram, optionally prefixing all logical paths.

When logical_prefix is provided, all ingested file paths become: {logical_prefix}/{relative_path_from_dir}.

pub fn ingest_file<P: AsRef<Path>>( &mut self, file_path: P, logical_path: String, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Ingest a single file into the engram with guaranteed reconstruction

This method encodes file data into sparse vectors and stores any necessary corrections to guarantee 100% bit-perfect reconstruction.

Correction Process

For each chunk:

1. Encode: chunk_data → SparseVec
2. Decode: SparseVec → decoded_data
3. Compare: chunk_data == decoded_data?
4. If different: store correction in CorrectionStore

Arguments

• file_path - Path to the file on disk
• logical_path - Path to use in the engram manifest
• verbose - Print progress information
• config - VSA encoding configuration

Returns

io::Result<()> indicating success or failure

pub fn add_file<P: AsRef<Path>>( &mut self, file_path: P, logical_path: String, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Add a new file to an existing engram (incremental update)

This method enables efficient incremental updates by adding a single file to an existing engram without requiring full re-ingestion. The new file’s chunks are bundled with the existing root vector using VSA’s associative bundle operation.

Algorithm

1. Encode new file into chunks (same as ingest_file)
2. Bundle each chunk with existing root: root_new = root_old ⊕ chunk
3. Add chunks to codebook with new chunk IDs
4. Update manifest with new file entry

Performance

• Time complexity: O(n) where n = number of chunks in new file
• Does not require reading or re-encoding existing files
• Suitable for production workflows with frequent additions

Arguments

• file_path - Path to the file on disk
• logical_path - Path to use in the engram manifest
• verbose - Print progress information
• config - VSA encoding configuration

Returns

io::Result<()> indicating success or failure

Examples

use embeddenator_fs::{EmbrFS, ReversibleVSAConfig};
use std::path::Path;

let mut fs = EmbrFS::new();
let config = ReversibleVSAConfig::default();

// Ingest initial dataset
fs.ingest_directory("./data", false, &config).unwrap();

// Later, add a new file without full re-ingestion
fs.add_file("./new_file.txt", "new_file.txt".to_string(), true, &config).unwrap();

pub fn remove_file(&mut self, logical_path: &str, verbose: bool) -> Result<()>

Remove a file from the engram (mark as deleted for incremental update)

This method marks a file as deleted in the manifest without modifying the root vector. This is because VSA bundling is a lossy operation and there’s no clean inverse. The chunks remain in the codebook but won’t be extracted.

Algorithm

1. Find file in manifest by logical path
2. Mark file entry as deleted
3. Chunks remain in codebook (for potential recovery or compaction)
4. File won’t appear in future extractions

Note on VSA Limitations

Bundle operation is associative but not invertible:

• (A ⊕ B) ⊕ C = A ⊕ (B ⊕ C) ✓ (can add)
• (A ⊕ B) ⊖ B ≠ A ✗ (can’t cleanly remove)

To truly remove chunks from the root, use compact() which rebuilds the engram without deleted files.

Arguments

• logical_path - Path of the file to remove
• verbose - Print progress information

Returns

io::Result<()> indicating success or failure

Examples

use embeddenator_fs::{EmbrFS, ReversibleVSAConfig};

let mut fs = EmbrFS::new();
let config = ReversibleVSAConfig::default();

fs.ingest_directory("./data", false, &config).unwrap();
fs.remove_file("old_file.txt", true).unwrap();
// File marked as deleted, won't be extracted

pub fn modify_file<P: AsRef<Path>>( &mut self, file_path: P, logical_path: String, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Modify an existing file in the engram (incremental update)

This method updates a file’s content by removing the old version and adding the new version. It’s equivalent to remove_file + add_file.

Algorithm

1. Mark old file as deleted
2. Re-encode new file content
3. Bundle new chunks with root
4. Add new file entry to manifest

Trade-offs

• Old chunks remain in codebook (use compact() to clean up)
• Root contains both old and new chunk contributions (slight noise)
• Fast operation, doesn’t require rebuilding entire engram

Arguments

• file_path - Path to the file on disk (new content)
• logical_path - Path of the file in the engram
• verbose - Print progress information
• config - VSA encoding configuration

Returns

io::Result<()> indicating success or failure

Examples

use embeddenator_fs::{EmbrFS, ReversibleVSAConfig};
use std::path::Path;

let mut fs = EmbrFS::new();
let config = ReversibleVSAConfig::default();

fs.ingest_directory("./data", false, &config).unwrap();

// Later, modify a file
fs.modify_file("./data/updated.txt", "data/updated.txt".to_string(), true, &config).unwrap();

pub fn compact( &mut self, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Compact the engram by rebuilding without deleted files

This operation rebuilds the engram from scratch, excluding all files marked as deleted. It’s the only way to truly remove old chunks from the root vector and codebook.

Algorithm

1. Create new empty engram
2. Re-bundle all non-deleted files
3. Reassign chunk IDs sequentially
4. Replace old engram with compacted version

Performance

• Time complexity: O(N) where N = total bytes of non-deleted files
• Expensive operation, run periodically (not after every deletion)
• Recommended: compact when deleted files exceed 20-30% of total

Benefits

• Reclaims space from deleted chunks
• Reduces root vector noise from obsolete data
• Resets chunk IDs to sequential order
• Maintains bit-perfect reconstruction of kept files

Arguments

• verbose - Print progress information
• config - VSA encoding configuration

Returns

io::Result<()> indicating success or failure

Examples

use embeddenator_fs::{EmbrFS, ReversibleVSAConfig};

let mut fs = EmbrFS::new();
let config = ReversibleVSAConfig::default();

fs.ingest_directory("./data", false, &config).unwrap();
fs.remove_file("old1.txt", false).unwrap();
fs.remove_file("old2.txt", false).unwrap();

// After many deletions, compact to reclaim space
fs.compact(true, &config).unwrap();

pub fn save_engram<P: AsRef<Path>>(&self, path: P) -> Result<()>

Save engram to file

pub fn load_engram<P: AsRef<Path>>(path: P) -> Result<Engram>

Load engram from file

pub fn save_manifest<P: AsRef<Path>>(&self, path: P) -> Result<()>

Save manifest to JSON file

pub fn load_manifest<P: AsRef<Path>>(path: P) -> Result<Manifest>

Load manifest from JSON file

pub fn load<P: AsRef<Path>, Q: AsRef<Path>>( engram_path: P, manifest_path: Q, ) -> Result<Self>

Load an EmbrFS from engram and manifest files

Automatically detects if the engram was created with holographic mode and sets up the appropriate encoder for extraction.

Arguments

• engram_path - Path to the engram file
• manifest_path - Path to the manifest JSON file

Returns

io::Result<EmbrFS> with the loaded engram and manifest

pub fn extract<P: AsRef<Path>>( engram: &Engram, manifest: &Manifest, output_dir: P, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Extract files from engram to directory with guaranteed reconstruction

This method guarantees 100% bit-perfect reconstruction by applying stored corrections after decoding each chunk.

Reconstruction Process

For each chunk:

1. Decode: SparseVec → decoded_data
2. Apply correction: decoded_data + correction → original_data
3. Verify: Hash matches stored hash (guaranteed by construction)

Arguments

• engram - The engram containing encoded data and corrections
• manifest - File metadata and chunk mappings
• output_dir - Directory to write extracted files
• verbose - Print progress information
• config - VSA decoding configuration

Returns

io::Result<()> indicating success or failure

pub fn extract_with_resonator<P: AsRef<Path>>( &self, output_dir: P, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Extract files using resonator-enhanced pattern completion with guaranteed reconstruction

Performs filesystem extraction with intelligent recovery capabilities powered by resonator networks. When chunks are missing from the codebook, the resonator attempts pattern completion to reconstruct the lost data, enabling extraction even from partially corrupted or incomplete engrams.

Reconstruction Guarantee

Even with resonator-assisted recovery, corrections are applied to guarantee bit-perfect reconstruction. The process is:

1. Try to get chunk from codebook
2. If missing, use resonator to recover approximate chunk
3. Apply correction from CorrectionStore
4. Result is guaranteed bit-perfect (if correction exists)

How it works

1. For each file chunk, check if it exists in the engram codebook
2. If missing, use the resonator to project a query vector onto known patterns
3. Apply stored corrections for guaranteed accuracy
4. Reconstruct the file from available and recovered chunks
5. If no resonator is configured, falls back to standard extraction

Why this matters

• Enables 100% reconstruction even with missing chunks
• Provides fault tolerance for distributed storage scenarios
• Supports hierarchical recovery at multiple levels of the storage stack
• Maintains data integrity through pattern-based completion

Arguments

• output_dir - Directory path where extracted files will be written
• verbose - Whether to print progress information during extraction
• config - VSA configuration for encoding/decoding

Returns

io::Result<()> indicating success or failure of the extraction operation

Examples

use embeddenator_fs::{EmbrFS, Resonator, ReversibleVSAConfig};
use std::path::Path;

let mut fs = EmbrFS::new();
let resonator = Resonator::new();
let config = ReversibleVSAConfig::default();
fs.set_resonator(resonator);

// Assuming fs has been populated with data...
let result = fs.extract_with_resonator("/tmp/output", true, &config);
assert!(result.is_ok());

pub fn bundle_hierarchically( &self, max_level_sparsity: usize, verbose: bool, _config: &ReversibleVSAConfig, ) -> Result<HierarchicalManifest>

Perform hierarchical bundling with path role binding and permutation tagging

Creates multi-level engram structures where path components are encoded using permutation operations to create distinct representations at each level. This enables efficient hierarchical retrieval and reconstruction.

How it works

1. Split file paths into components (e.g., “a/b/c.txt” → [“a”, “b”, “c.txt”])
2. For each level, apply permutation based on path component hash
3. Bundle representations level-by-level with sparsity control
4. Create sub-engrams for intermediate nodes

Why this matters

• Enables scalable hierarchical storage beyond flat bundling limits
• Path-based retrieval without full engram traversal
• Maintains semantic relationships through permutation encoding
• Supports efficient partial reconstruction

Arguments

• max_level_sparsity - Maximum non-zero elements per level bundle
• verbose - Whether to print progress information

Returns

HierarchicalManifest describing the multi-level structure

Examples

use embeddenator_fs::{EmbrFS, ReversibleVSAConfig};

let fs = EmbrFS::new();
let config = ReversibleVSAConfig::default();
// Assuming files have been ingested...

let hierarchical = fs.bundle_hierarchically(500, false, &config);
assert!(hierarchical.is_ok());

pub fn bundle_hierarchically_with_options( &self, max_level_sparsity: usize, max_chunks_per_node: Option<usize>, verbose: bool, _config: &ReversibleVSAConfig, ) -> Result<HierarchicalManifest>

Like bundle_hierarchically, but supports an optional deterministic cap on chunk_ids per node.

If max_chunks_per_node is set and a node would exceed that many chunk_ids, the node becomes a router with empty chunk_ids, and deterministic shard children are created each containing a bounded subset of chunk_ids.

pub fn extract_hierarchically<P: AsRef<Path>>( &self, hierarchical: &HierarchicalManifest, output_dir: P, verbose: bool, config: &ReversibleVSAConfig, ) -> Result<()>

Extract files from hierarchical manifest with manifest-guided traversal

Performs hierarchical extraction by traversing the manifest levels and reconstructing files from sub-engrams. This enables efficient extraction from complex hierarchical structures without loading the entire engram.

How it works

1. Traverse manifest levels from root to leaves
2. For each level, locate relevant sub-engrams
3. Reconstruct file chunks using inverse permutation operations
4. Assemble complete files from hierarchical components

Why this matters

• Enables partial extraction from large hierarchical datasets
• Maintains bit-perfect reconstruction accuracy
• Supports efficient path-based queries and retrieval
• Scales to complex directory structures

Arguments

• hierarchical - The hierarchical manifest to extract from
• output_dir - Directory path where extracted files will be written
• verbose - Whether to print progress information during extraction

Returns

io::Result<()> indicating success or failure of the hierarchical extraction

Examples

use embeddenator_fs::{EmbrFS, ReversibleVSAConfig};

let fs = EmbrFS::new();
let config = ReversibleVSAConfig::default();
// Assuming hierarchical manifest was created...
// let hierarchical = fs.bundle_hierarchically(500, true).unwrap();

// fs.extract_hierarchically(&hierarchical, "/tmp/output", true, &config)?;

Trait Implementations

impl Default for EmbrFS

fn default() -> Self

Returns the “default value” for a type.