Module ingest_arrow 

Ingest Parquet and Arrow IPC files into Hyper tables.

These are “Tier 1” (exact schema) ingest paths: types come directly from the file’s embedded schema, so there is no guessing.

Parquet path: native hyperd read

For Parquet files we let hyperd read the file directly via the external(path, FORMAT => 'parquet') table function. The MCP issues a single CREATE TABLE ... AS SELECT * FROM external(...) (replace mode) or INSERT INTO ... SELECT * FROM external(...) (append mode). This pushes the entire read into hyperd’s C++ Parquet reader — no Rust-side Arrow decode, no per-row INSERTs, and native support for the full set of compressions and encodings hyperd understands (including Snappy, which isn’t compiled into the Rust parquet crate in this build).

Benchmarked at ~3.5M rows/sec on TPCH 1GB lineitem (6M rows / 360 MB in ~1.7s) — roughly three orders of magnitude faster than the previous row-by-row path.

We still open the Parquet footer in Rust to infer column types — that’s cheap (footer-only metadata read) and lets us report the schema in IngestResult and apply user-supplied schema overrides by wrapping each overridden column in an explicit ::TYPE cast inside the SELECT projection.

Arrow IPC path: binary COPY via ArrowInserter

For Arrow IPC (Feather) files we use hyperdb-api’s ArrowInserter / AsyncArrowInserter, which push RecordBatches straight through Hyper’s binary COPY protocol as an Arrow IPC Stream. No text encoding, no per-row SQL — the batches we decode from the IPC File format are re-framed as IPC Stream messages and sent wholesale to the server.

User-supplied schema overrides are applied by pre-creating the target table with the overridden column types and letting Hyper coerce the incoming values on insert. This avoids a second Rust-side transform pass over the data.

Functions

arrow_schema_to_columns

Convert an Arrow schema to our internal ColumnSchema representation, preserving nullability from the Arrow field metadata.

ingest_arrow_ipc_file

Ingest an Arrow IPC (Feather) file into a Hyper table via the binary COPY protocol.

ingest_arrow_ipc_file_async

Async twin of ingest_arrow_ipc_file. File I/O and Arrow decode run on the blocking pool; the COPY stream is driven on the async connection. hyperdb_api::AsyncArrowInserter accepts raw IPC Stream bytes, so we serialize the decoded batches into one buffer and send it in a single insert_data call.

ingest_parquet_file

Ingest a Parquet file into a Hyper table using hyperd’s native parquet reader via CREATE TABLE ... AS SELECT * FROM external(...).

ingest_parquet_file_async

Async twin of ingest_parquet_file. Uses an owned AsyncConnection so it can run on a pooled connection from the watcher without touching the engine’s primary sync connection. Footer metadata is read on the blocking pool (tiny read, but file I/O).