Skip to main content

Module decoder

Module decoder 

Source

Enums§

RecordBoundary
How records are separated in the input.
SplitMode
Mode for chunk planning: parallel and streaming have different optima.

Constants§

MAX_SPLIT_CHUNKS
Maximum number of split chunks. Above this, scheduling overhead dominates.
MIN_CHUNK_BYTES
Minimum chunk size in bytes. Sub-MB chunks collapse throughput due to per-chunk fixed cost (thread dispatch, cache cold start). Measured: 100 MB at par128 (0.78 MB chunks) = 2,265 MB/s vs par16 (6.25 MB) = 3,735.

Traits§

ColumnarSink
Sink for decoder events. The decoder calls begin_row / put_field / end_row for each record.
RecordParser
Format-specific parser: turns a byte chunk into field/value events sent to a ColumnarSink.
SkipRegionFinder
Finds byte ranges (comments, CDATA, quoted fields, string literals) where a candidate record boundary must be rejected. The engine calls this during split-point discovery to avoid splitting inside comments or strings.
Splitter
Format-specific splitter: decides where it is safe to divide an input byte stream into independent chunks.

Functions§

find_next_record_boundary
Scan forward from from to find the next record boundary.
in_skip_region
Check whether byte position at falls inside a skip region by scanning backward up to finder.window() bytes, looking for an unclosed opener.
is_continued_newline
Check whether the newline at pos is preceded by a continuation character.
plan_chunk_count
plan_chunk_count_with using the default MIN_CHUNK_BYTES floor.
plan_chunk_count_with
Plan the number of chunks given input size, thread count, and mode. Returns a value in [threads, MAX_SPLIT_CHUNKS] with a floor of min_chunk_bytes (default MIN_CHUNK_BYTES via plan_chunk_count).
split_points_to_ranges
Convert split-point offsets returned by a Splitter into non-empty Range<usize> chunks.