Skip to main content

Splitter

Trait Splitter 

Source
pub trait Splitter: Send + Sync {
    // Required methods
    fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>;
    fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize;

    // Provided methods
    fn skip_regions(&self) -> Option<&dyn SkipRegionFinder> { ... }
    fn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize> { ... }
}
Expand description

Format-specific splitter: decides where it is safe to divide an input byte stream into independent chunks.

The only required method is next_record_start; adapters provide the record boundary logic, and the engine handles chunk planning, skip-region rejection, and deduplication.

Required Methods§

Source

fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>

Return the next record boundary at or after from. Must return a position where a record starts, or None if no more records exist.

Source

fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize

Estimate the average bytes per record from a sample of the input.

Provided Methods§

Source

fn skip_regions(&self) -> Option<&dyn SkipRegionFinder>

Optional: byte ranges where a candidate boundary must be rejected. See in_skip_region.

Source

fn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize>

Find split points. Do not override without a measured reason.

Default implementation:

  1. Nominal offsets at bytes.len() * i / n for i in 1..n.
  2. par_iter over nominals (rayon), each calling next_record_start.
  3. Reject candidates inside skip regions via in_skip_region.
  4. Dedup, sort, prepend 0.
  5. Apply chunk floor via plan_chunk_count.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§