Haagenti Compressed Tensor (HCT) Format

High-performance compressed tensor storage for neural network weights, with HoloTensor holographic compression support for progressive loading.

Overview

HCT provides two complementary storage modes:

Standard HCT: Block-compressed tensor storage with random access
HoloTensor: Holographic compression enabling progressive reconstruction

Standard HCT Format

Block-based compression with LZ4 or Zstd for fast random access:

use haagenti_hct::{HctWriter, HctReader, CompressionAlgorithm, DType};
use std::fs::File;

// Write compressed tensor
let mut writer = HctWriter::new(
    File::create("weights.hct")?,
    CompressionAlgorithm::Zstd,
    DType::F16,
    &[4096, 4096],
)?;
writer.write_data(&weight_data)?;
writer.finish()?;

// Read tensor
let mut reader = HctReader::open("weights.hct")?;
let data = reader.read_all()?;

HoloTensor Format

Holographic compression enables progressive reconstruction from partial data:

use haagenti_hct::{
    HoloTensorEncoder, HoloTensorDecoder,
    HolographicEncoding, DType,
};

// Encode with spectral holography (8 fragments)
let encoder = HoloTensorEncoder::new(HolographicEncoding::Spectral)
    .with_fragments(8);
let (header, fragments) = encoder.encode_1d(&weights)?;

// Reconstruct from partial fragments (any 4 of 8 for ~90% quality)
let mut decoder = HoloTensorDecoder::new(header);
decoder.add_fragment(fragments[0].clone())?;
decoder.add_fragment(fragments[3].clone())?;
decoder.add_fragment(fragments[5].clone())?;
decoder.add_fragment(fragments[7].clone())?;

let approx_data = decoder.reconstruct()?;

Encoding Schemes

Scheme	Best For	Min Quality	Progressive
Spectral (DCT)	Dense MLP weights	60%	Smooth curve
Random Projection	High-dimensional	10%	Linear curve
Low-Rank Distributed	Attention layers	30%	Sharp knee

Feature Flags