Skip to main content

Module build

hexz_cli::cmd::data

Module build 

Source
Expand description

Build archive from source with profile-based optimization.

This command provides a high-level interface for creating Hexz snapshots with domain-specific optimizations. Unlike the low-level pack command, build uses predefined profiles that automatically select compression algorithms, block sizes, and dictionary training settings optimized for different workload types.

§Build Profiles

Profiles configure compression and chunking parameters based on workload characteristics:

Generic Profile (Default):

  • Compression: LZ4 (fast, general-purpose)
  • Block size: 64 KiB (balanced for most workloads)
  • Dictionary training: Disabled (minimal overhead)
  • Use case: General operating system images, file servers, development VMs

EDA Profile (Electronic Design Automation):

  • Compression: Zstd level 3 (high ratio for large design files)
  • Block size: 128 KiB (optimized for large CAD files and netlists)
  • Dictionary training: Enabled (learns patterns from design data)
  • Use case: ASIC/FPGA design environments with large binary databases

Embedded Profile:

  • Compression: LZ4 (minimal CPU overhead for resource-constrained targets)
  • Block size: 16 KiB (smaller blocks reduce memory pressure)
  • Dictionary training: Disabled (reduces snapshot creation time)
  • Use case: IoT devices, embedded Linux systems, edge computing

ML Profile (Machine Learning):

  • Compression: Zstd level 3 (handles large model files efficiently)
  • Block size: 256 KiB (optimized for model weights and training data)
  • Dictionary training: Enabled (learns patterns from tensor data)
  • Use case: ML training environments, GPU workstations, Jupyter notebooks

§Build Profile Effects

§Compression Algorithm Selection

  • LZ4: Provides 2-3x compression at 500+ MB/s, ideal for fast boot times
  • Zstd: Provides 3-5x compression at 200+ MB/s, ideal for storage efficiency

§Block Size Selection

Smaller blocks (16-64 KiB):

  • Lower memory usage during decompression
  • Finer-grained deduplication
  • Higher index overhead

Larger blocks (128-256 KiB):

  • Better compression ratios (larger context window)
  • Reduced index size
  • Higher decompression memory requirements

§Dictionary Training

When enabled, samples ~1000 blocks and trains a Zstd dictionary:

  • Improves compression ratio by 10-30% for repetitive data
  • Adds 2-5 seconds to snapshot creation time
  • Dictionary size: ~110 KiB stored in snapshot header

§Use Cases

  • VM Image Creation: Build bootable snapshots from disk images
  • Reproducible Environments: Create snapshots with consistent compression settings
  • Workload Optimization: Select profiles matched to application characteristics
  • CI/CD Pipelines: Automate snapshot creation with profile presets

§Common Usage Patterns

# Build generic snapshot from disk image
hexz build --source disk.img --output snapshot.st

# Build EDA workstation with optimal compression
hexz build --source eda-vm.img --output eda.st --profile eda

# Build ML environment with encryption
hexz build --source ml.img --output ml.st --profile ml --encrypt

# Build with content-defined chunking for deduplication
hexz build --source app.img --output app.st --cdc

Functions§

run
Executes the build command to create a snapshot using profile-based settings.