pub trait Splitter: Send + Sync {
// Required methods
fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>;
fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize;
// Provided methods
fn record_boundary(&self) -> RecordBoundary { ... }
fn continuation_char(&self) -> Option<u8> { ... }
fn comment_prefixes(&self) -> &[&[u8]] { ... }
fn skip_regions(&self) -> Option<&dyn SkipRegionFinder> { ... }
fn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize> { ... }
fn find_split_points_with(
&self,
bytes: &[u8],
max_chunks: usize,
min_chunk_bytes: usize,
) -> Vec<usize> { ... }
}Expand description
Format-specific splitter: decides where it is safe to divide an input byte stream into independent chunks.
The only required method is next_record_start; adapters provide the
record boundary logic, and the engine handles chunk planning, skip-region
rejection, and deduplication.
For multi-line formats, implement the declarative methods
(record_boundary,
continuation_char,
comment_prefixes) and call
find_next_record_boundary from your next_record_start:
use rypipe_core::{find_next_record_boundary, RecordBoundary, Splitter};
struct Lines;
impl Splitter for Lines {
fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize> {
find_next_record_boundary(bytes, from, self.continuation_char(), self.comment_prefixes(), self.record_boundary() == RecordBoundary::BlankLine)
}
fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize { sample.len().max(1) }
}
assert_eq!(Lines.next_record_start(b"one\ntwo\n", 0), Some(4));Required Methods§
Sourcefn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>
fn next_record_start(&self, bytes: &[u8], from: usize) -> Option<usize>
Return the next record boundary at or after from. Must return a
position where a record starts, or None if no more records exist.
Sourcefn estimate_bytes_per_row(&self, sample: &[u8]) -> usize
fn estimate_bytes_per_row(&self, sample: &[u8]) -> usize
Estimate the average bytes per record from a sample of the input.
Provided Methods§
Sourcefn record_boundary(&self) -> RecordBoundary
fn record_boundary(&self) -> RecordBoundary
How records are separated. Default: RecordBoundary::Line (every
newline is a boundary). Override to RecordBoundary::BlankLine for
multi-line formats where blank lines separate records.
Sourcefn continuation_char(&self) -> Option<u8>
fn continuation_char(&self) -> Option<u8>
Character that continues a record across lines (e.g. b'\\' for
properties files). When set, newlines preceded by this character are
NOT record boundaries. Default: None (no continuations).
Sourcefn comment_prefixes(&self) -> &[&[u8]]
fn comment_prefixes(&self) -> &[&[u8]]
Line prefixes to skip (e.g. &[b"#", b"!"] for properties files).
Lines starting with any of these are ignored during splitting.
Default: empty (no comments).
Sourcefn skip_regions(&self) -> Option<&dyn SkipRegionFinder>
fn skip_regions(&self) -> Option<&dyn SkipRegionFinder>
Optional: byte ranges where a candidate boundary must be rejected.
See in_skip_region.
Sourcefn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize>
fn find_split_points(&self, bytes: &[u8], max_chunks: usize) -> Vec<usize>
Find split points. Do not override without a measured reason.
Returns a sorted list of byte offsets where it is safe to split the input into independent chunks. The return value must satisfy:
- Sorted ascending, no duplicates.
- First element is
0, last element isbytes.len(). - All values are
<= bytes.len().
Default implementation:
- Nominal offsets at
bytes.len() * i / nfori in 1..n. - Scan each nominal offset with
next_record_start. - Reject candidates inside skip regions via
in_skip_region. - Dedup, sort, prepend 0.
- Apply chunk floor via
plan_chunk_count.
This is the default-floor entry point. Adapters that override it keep
working; engines call Splitter::find_split_points_with instead, and
that defers to this override at the default floor.
Sourcefn find_split_points_with(
&self,
bytes: &[u8],
max_chunks: usize,
min_chunk_bytes: usize,
) -> Vec<usize>
fn find_split_points_with( &self, bytes: &[u8], max_chunks: usize, min_chunk_bytes: usize, ) -> Vec<usize>
Find split points with an explicit chunk-size floor.
The floor is opt-in: lowering it yields more, smaller chunks (better
load balancing on many or small cores, at more scheduling overhead);
raising it yields fewer, larger chunks. Default is MIN_CHUNK_BYTES.
At the default floor this calls Splitter::find_split_points, so an
adapter’s override is honored; a non-default floor uses the engine’s
default planning (override find_split_points_with to customize it).
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".