Crate tensor_blob 

TensorBlob - Module 11 of Neumann

S3-style object storage for large artifacts using content-addressable chunked storage with tensor-native metadata.

Features

• Content-Addressable: Chunks keyed by SHA-256 hash for automatic deduplication
• Tensor-Native: Metadata participates in graph/relational/vector queries
• Streaming: Large files never fully in memory
• Linked Artifacts: Objects connect to entities via graph edges
• Garbage Collected: Orphaned chunks cleaned up automatically
• Async-First: All I/O operations are async

Example

use tensor_blob::{BlobStore, BlobConfig, PutOptions};

let store = BlobStore::new(TensorStore::new(), BlobConfig::default()).await?;

// Store an artifact
let artifact_id = store.put(
    "report.pdf",
    &file_bytes,
    PutOptions::new()
        .with_created_by("user:alice")
        .with_tag("quarterly"),
).await?;

// Retrieve it
let data = store.get(&artifact_id).await?;

Structs

ArtifactMetadata

Metadata about a stored artifact.

BlobConfig

Configuration for the blob store.

BlobReader

Streaming reader for downloading artifacts.

BlobStats

Statistics about blob storage.

BlobStore

S3-style blob store with content-addressable chunked storage.

BlobWriter

Streaming writer for uploading artifacts.

Chunk

A content-addressed chunk of data.

Chunker

Chunker for splitting data into content-addressable chunks.

GarbageCollector

Background garbage collector for orphaned chunks.

GcConfig

Configuration for garbage collection.

GcStats

Result from garbage collection.

MetadataUpdates

Updates to apply to artifact metadata.

PutOptions

Options for storing a new artifact.

RepairStats

Result from repair operation.

SimilarArtifact

Result from similarity search.

StreamingHasher

A streaming hasher for computing checksums incrementally.

Enums

BlobError

Functions

check_chunks_exist

Check if all chunks for an artifact exist.

compute_hash

Compute SHA-256 hash of data.

compute_hash_streaming

Compute SHA-256 hash of multiple data segments.

find_orphaned_chunks

Find all chunks not referenced by any artifact.

verify_chunk

Verify a single chunk’s integrity by checking its content hash matches its key.

Type Aliases

Result