Module block_storage

sochdb_core

Module block_storage 

Source
Expand description

File Data Block Storage (Task 12)

Integrates PayloadStore as SochFS data block backend:

  • O(1) append-only writes (no LSM compaction overhead)
  • Transparent compression (LZ4/ZSTD based on content type)
  • Block reference tracking for garbage collection

§Block Storage Architecture

┌─────────────────────────────────────────────────────────┐
│                      Inode Table                        │
│  inode_id=7, blocks=[(off=0, len=4096, cmp=LZ4),       │
│                      (off=4096, len=2048, cmp=ZSTD)]   │
└─────────────────────────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────┐
│                    PayloadStore                         │
│  Offset 0:    [LZ4 header][compressed block 1]         │
│  Offset 4096: [ZSTD header][compressed block 2]        │
└─────────────────────────────────────────────────────────┘

§Architectural Improvements (Production Tasks)

§Task 1: Fixed-Layout BlockHeader

Uses deterministic 17-byte fixed layout with explicit little-endian encoding:

Offset  Size  Field            Type
0       4     magic            [u8; 4] = "TBLK"
4       1     compression      u8
5       4     original_size    u32 (LE)
9       4     compressed_size  u32 (LE)
13      4     checksum         u32 (LE, CRC32)
Total: 17 bytes

§Task 5: Error Propagation

All serialization methods return Result<T, SochDBError> instead of using unwrap_or_default() which silently swallows errors.

Structs§

BlockHeader
Block header in payload store (Task 1: Fixed 17-byte layout)
BlockRef
Block reference stored in inode
BlockStore
File block storage backed by append-only store
BlockStoreStats
Block store statistics
DurableBlockStore
Durable block store with WAL-backed persistence
DurableBlockStoreStats
Durable block store statistics
FileBlockManager
File block manager - higher level interface for file I/O
RecoveryStats
Recovery statistics
WalIterator
Iterator over WAL records
WalReader
WAL reader for recovery
WalRecordHeader
WAL record header (fixed 33-byte layout)
WalWriter
WAL writer for durable writes

Enums§

BlockCompression
Compression type for blocks
WalRecordType
WAL record types

Constants§

DEFAULT_BLOCK_SIZE
Default block size (4KB)
MAX_BLOCK_SIZE
Maximum block size (1MB)

Functions§

is_compressible
Check if data is likely compressible
is_json_content
Check if data looks like JSON
is_soch_content
Check if data looks like TOON format