Crate seq_chunking

SeqChunking

A Rust library for sequence-based data chunking using slope detection algorithms.

This library provides efficient algorithms for dividing data streams into chunks based on byte sequence patterns (increasing or decreasing slopes). It’s particularly useful for content-defined chunking applications.

Quick Start

use seq_chunking::{SeqChunking, ChunkingConfig};

// Create a chunker with default settings
let chunker = SeqChunking::new();

// Or with custom configuration
let config = ChunkingConfig::builder()
    .seq_threshold(10)
    .min_block_size(2048)
    .build().expect("Failed to build ChunkingConfig");
let chunker = SeqChunking::from_config(config);

// Chunk some data
let data = b"your data here";
let chunks: Vec<_> = chunker.chunk_all(data).collect();

Re-exports

pub use config::ChunkingConfig;

pub use config::SeqOpMode;

pub use chunker::SeqChunking;

pub use chunker::Chunk;

pub use chunker::ChunkIterator;

pub use error::ChunkingError;

pub use error::Result;

Modules

chunker

Core chunking implementation.

config

Configuration module for sequence-based chunking.

error

Error handling for the chunking library.

utils

Utility functions for the chunking library.

Constants

DEFAULT_AVG_BLOCK_SIZE

Default average block size

DEFAULT_JUMP_SIZE

Default jump size when trigger is hit

DEFAULT_JUMP_TRIGGER

Default jump trigger count

DEFAULT_MAX_BLOCK_SIZE

Default maximum block size

DEFAULT_MIN_BLOCK_SIZE

Default minimum block size

DEFAULT_SEQ_THRESHOLD

Default sequence length threshold