Module compaction 

Compaction Module for Object Storage

This module provides segment management and compaction for object storage backends. It helps maintain storage efficiency by:

• Organizing vectors into segments for batch I/O
• Merging small segments into larger ones
• Garbage collecting deleted vectors
• Running background compaction based on configurable thresholds

Architecture

┌─────────────────────────────────────────────────────┐
│                  CompactionManager                   │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │
│  │  Segment 1  │  │  Segment 2  │  │  Segment N  │ │
│  │  (1000 vec) │  │  (500 vec)  │  │  (750 vec)  │ │
│  └─────────────┘  └─────────────┘  └─────────────┘ │
│         │                │                │         │
│         └────────────────┼────────────────┘         │
│                          ▼                          │
│                   Compaction Job                    │
│                          │                          │
│                          ▼                          │
│              ┌─────────────────────┐               │
│              │   Merged Segment    │               │
│              │    (2250 vec)       │               │
│              └─────────────────────┘               │
└─────────────────────────────────────────────────────┘

Structs

CompactionConfig

Configuration for compaction operations

CompactionJob

Compaction job status

CompactionManager

Global compaction manager for all namespaces

CompactionResult

Result of a compaction operation

CompactionStats

Statistics for compaction operations (atomic)

GlobalCompactionStats

Global compaction statistics

NamespaceSegmentManager

Manages segments for a namespace

Segment

A segment containing vectors

SegmentMetadata

Metadata for a segment

SegmentStats

Statistics for a namespace’s segments

Enums

CompactionJobState

State of a compaction job

SegmentState

Segment state

Type Aliases

SegmentId

Unique identifier for a segment