overlap-chunk

A Rust library for splitting text into chunks of specified size with adjustable overlap percentage.

Features

Current Features

Basic functionality to split text into chunks of specified size
Option to adjust the overlap percentage between chunks
Command-line interface for easy text processing

Future Features

Chunking that respects word boundaries and sentence boundaries
Support for multilingual text
Support for streaming input

Library Usage

use overlap_chunk::ChunkOptions;
use overlap_chunk::chunk_text;

fn main() {
    let text = "This is a test text. We will split this long text into smaller chunks.";
    
    // Chunk splitting with default options (no overlap)
    let chunks = chunk_text(text, 10, None);
    println!("{:?}", chunks);
    
    // Chunk splitting with overlap (50% overlap)
    let options = ChunkOptions {
        overlap_percentage: 50,
        ..Default::default()
    };
    let chunks_with_overlap = chunk_text(text, 10, Some(options));
    println!("{:?}", chunks_with_overlap);
}

Command Line Usage

The library includes a command-line interface for processing text files:

Usage: overlap-chunk [OPTIONS] [FILE]
  If no file is specified, read from standard input

Options:
  -h, --help              Display this help message
  -s, --size SIZE         Specify chunk size (default: 100)
  -o, --overlap PERCENT   Specify overlap percentage between 0 and 90 (default: 0)

Examples

Process a file with default settings:

overlap-chunk myfile.txt

Process a file with custom chunk size and overlap:

overlap-chunk -s 50 -o 30 myfile.txt

Process standard input:

cat myfile.txt | overlap-chunk -s 50

License

MIT License

overlap-chunk 0.0.2