Skip to main content Module decoder Copy item path Source SplitMode Mode for chunk planning: parallel and streaming have different optima. MAX_SPLIT_CHUNKS Maximum number of split chunks. Above this, scheduling overhead dominates. MIN_CHUNK_BYTES Minimum chunk size in bytes. Sub-MB chunks collapse throughput due to
per-chunk fixed cost (thread dispatch, cache cold start). Measured:
100 MB at par128 (0.78 MB chunks) = 2,265 MB/s vs par16 (6.25 MB) = 3,735. ColumnarSink Sink for decoder events. The decoder calls begin_row / put_field /
end_row for each record. RecordParser Format-specific parser: turns a byte chunk into field/value events sent to
a ColumnarSink. SkipRegionFinder Finds byte ranges (comments, CDATA, quoted fields, string literals) where
a candidate record boundary must be rejected. The engine calls this
during split-point discovery to avoid splitting inside comments or strings. Splitter Format-specific splitter: decides where it is safe to divide an input byte
stream into independent chunks. in_skip_region Check whether byte position at falls inside a skip region by scanning
backward up to finder.window() bytes, looking for an unclosed opener. plan_chunk_count Plan the number of chunks given input size, thread count, and mode.
Returns a value in [threads, MAX_SPLIT_CHUNKS] with a 2 MiB floor. split_points_to_ranges Convert split-point offsets returned by a Splitter into non-empty
Range<usize> chunks.