Skip to main content

Splitter

Trait Splitter 

Source
pub trait Splitter: Send + Sync {
    // Required methods
    fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>;
    fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize;

    // Provided methods
    fn record_boundary(&self) -> RecordBoundary { ... }
    fn continuation_char(&self) -> Option<u8> { ... }
    fn comment_prefixes(&self) -> &[&[u8]] { ... }
    fn skip_regions(&self) -> Option<&dyn SkipRegionFinder> { ... }
    fn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize> { ... }
    fn find_split_points_with(
        &self,
        bytes: &[u8],
        max_chunks: usize,
        min_chunk_bytes: usize,
    ) -> Vec<usize> { ... }
}
Expand description

Format-specific splitter: decides where it is safe to divide an input byte stream into independent chunks.

The only required method is next_record_start; adapters provide the record boundary logic, and the engine handles chunk planning, skip-region rejection, and deduplication.

For multi-line formats, implement the declarative methods (record_boundary, continuation_char, comment_prefixes) and call find_next_record_boundary from your next_record_start:

use rypipe_core::{find_next_record_boundary, RecordBoundary, Splitter};
struct Lines;
impl Splitter for Lines {
    fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize> {
        find_next_record_boundary(bytes, from, self.continuation_char(), self.comment_prefixes(), self.record_boundary() == RecordBoundary::BlankLine)
    }
    fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize { sample.len().max(1) }
}
assert_eq!(Lines.next_record_start(b"one\ntwo\n", 0), Some(4));

Required Methods§

Source

fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>

Return the next record boundary at or after from. Must return a position where a record starts, or None if no more records exist.

Source

fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize

Estimate the average bytes per record from a sample of the input.

Provided Methods§

Source

fn record_boundary(&self) -> RecordBoundary

How records are separated. Default: RecordBoundary::Line (every newline is a boundary). Override to RecordBoundary::BlankLine for multi-line formats where blank lines separate records.

Source

fn continuation_char(&self) -> Option<u8>

Character that continues a record across lines (e.g. b'\\' for properties files). When set, newlines preceded by this character are NOT record boundaries. Default: None (no continuations).

Source

fn comment_prefixes(&self) -> &[&[u8]]

Line prefixes to skip (e.g. &[b"#", b"!"] for properties files). Lines starting with any of these are ignored during splitting. Default: empty (no comments).

Source

fn skip_regions(&self) -> Option<&dyn SkipRegionFinder>

Optional: byte ranges where a candidate boundary must be rejected. See in_skip_region.

Source

fn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize>

Find split points. Do not override without a measured reason.

Returns a sorted list of byte offsets where it is safe to split the input into independent chunks. The return value must satisfy:

  • Sorted ascending, no duplicates.
  • First element is 0, last element is bytes.len().
  • All values are <= bytes.len().

Default implementation:

  1. Nominal offsets at bytes.len() * i / n for i in 1..n.
  2. Scan each nominal offset with next_record_start.
  3. Reject candidates inside skip regions via in_skip_region.
  4. Dedup, sort, prepend 0.
  5. Apply chunk floor via plan_chunk_count.

This is the default-floor entry point. Adapters that override it keep working; engines call Splitter::find_split_points_with instead, and that defers to this override at the default floor.

Source

fn find_split_points_with( &self, bytes: &[u8], max_chunks: usize, min_chunk_bytes: usize, ) -> Vec<usize>

Find split points with an explicit chunk-size floor.

The floor is opt-in: lowering it yields more, smaller chunks (better load balancing on many or small cores, at more scheduling overhead); raising it yields fewer, larger chunks. Default is MIN_CHUNK_BYTES.

At the default floor this calls Splitter::find_split_points, so an adapter’s override is honored; a non-default floor uses the engine’s default planning (override find_split_points_with to customize it).

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§