Module native 

Native byte-passthrough fast-load (#633) — the third capability-negotiated transfer mechanism, alongside the Arrow columnar path (RFC 0002 / #375) and staged bulk load (#528).

When a source can emit its records as raw bytes in a wire format (NativeFormat) and a sink can bulk-load that exact format directly, and no Value-shaped stage sits between them, the pipeline streams the bytes source → sink without ever materializing serde_json::Value or an Arrow RecordBatch — the minimum-memory, minimum-CPU path.

This is the byte analogue of the columnar path: columnar is for typed columnar sources (parquet/delta) → typed sinks; byte-passthrough is for format-matched wire-byte pairs (a bulk-export CSV → BigQuery CSV load; an S3 .jsonl object → BigQuery NDJSON load) with no object-store hop, where the destination does the CSV/JSON → typed-column casting.

Like the other two mechanisms it is opt-in and additive: a source advertises formats via Source::native_output_formats and a sink advertises mechanisms via Sink::native_load_capabilities. The pipeline uses the path only when plan_native_transfer finds a sink capability whose format the source offers and whose write modes cover the run — after the planner’s own unconditional gates (no transforms, no quality/contract/masking pass, no DLQ, at-least-once only) have passed. The “no transformation between” gate is additionally enforced structurally — a TransformingSource does not override native_output_formats, so any attached transform makes the wrapped source advertise no formats and the fast path falls through to the Value path.

The core types carry no heavy dependencies (byte payloads are Vec<u8> / boxed byte streams, matching crate::staging), so the negotiation is always compiled — it is a first-class part of the Source/Sink contract, not a Cargo feature. Individual connector implementations may still gate their stream_native / load_native bodies behind their own features.

Structs

CsvDialect

CSV specifics carried on a NativeBatch (ignored for other formats). The sink must be able to honor the source’s dialect, or refuse the match.

NativeBatch

One native-format byte batch handed from a source’s stream_native to a sink’s load_native.

NativeLoadCapability

An efficient native-load mechanism a sink offers. A sink returns one of these per mechanism it supports from Sink::native_load_capabilities.

NativeLoadContext

Per-call context the pipeline hands each load_native invocation.

NativePlan

The negotiated native transfer: which format to move and which sink mechanism loads it.

NativePlanInputs

Inputs to the pure plan_native_transfer negotiation.

Enums

NativeFormat

A wire format that can move source → sink as raw bytes, bypassing per-record serde_json::Value (and Arrow) materialization. This is the negotiation key: a source advertises the formats it can emit and a sink the formats it can load, and plan_native_transfer matches them.

NativePayload

The bytes of one native batch — either fully buffered or a byte stream.

Functions

plan_native_transfer

Decide whether a native byte-passthrough fast path applies, and which.