Skip to main content

Module decoder

Module decoder 

Source

Enums§

SplitMode
Mode for chunk planning: parallel and streaming have different optima.

Constants§

MAX_SPLIT_CHUNKS
Maximum number of split chunks. Above this, scheduling overhead dominates.
MIN_CHUNK_BYTES
Minimum chunk size in bytes. Sub-MB chunks collapse throughput due to per-chunk fixed cost (thread dispatch, cache cold start). Measured: 100 MB at par128 (0.78 MB chunks) = 2,265 MB/s vs par16 (6.25 MB) = 3,735.

Traits§

ColumnarSink
Sink for decoder events. The decoder calls begin_row / put_field / end_row for each record.
RecordParser
Format-specific parser: turns a byte chunk into field/value events sent to a ColumnarSink.
SkipRegionFinder
Finds byte ranges (comments, CDATA, quoted fields, string literals) where a candidate record boundary must be rejected. The engine calls this during split-point discovery to avoid splitting inside comments or strings.
Splitter
Format-specific splitter: decides where it is safe to divide an input byte stream into independent chunks.

Functions§

in_skip_region
Check whether byte position at falls inside a skip region by scanning backward up to finder.window() bytes, looking for an unclosed opener.
plan_chunk_count
Plan the number of chunks given input size, thread count, and mode. Returns a value in [threads, MAX_SPLIT_CHUNKS] with a 2 MiB floor.
split_points_to_ranges
Convert split-point offsets returned by a Splitter into non-empty Range<usize> chunks.