pub async fn export_cypher_chunked(
executor: &mut FalkorExecutor,
cypher: &str,
dir: impl AsRef<Path>,
file_type: FileType,
chunk_rows: usize,
) -> Result<ExportReport>Expand description
Run cypher and write the result as multiple GraphAr chunk files —
chunk0.<ext>, chunk1.<ext>, … — under dir, each holding at most
chunk_rows rows. This is the chunked export for huge results: instead of
one monolithic file it lands the result the way GraphAr naturally tiles a
vertex/edge collection (one file per row-range), so a VertexReader (or a
plain read_chunk per file) can stream the chunks back without ever holding
the whole result in memory.
Returns an ExportReport whose chunks lists the files written in order.
§Streaming honesty — why this is slice-and-write, not pull-stream
FalkorDB’s GRAPH.QUERY is a single RESP request/response: the server
serializes the entire result set into one reply with no server-side
cursor, and FalkorExecutor::query_auto therefore materializes it as one
in-memory RecordBatch. There is no API to pull
the result in pages, so we cannot bound the read side below the full
result. (FalkorDB ≥ 4 exposes GRAPH.QUERY … TIMEOUT/result-set limits but
no streaming cursor over RESP.)
What this does bound is the write side and downstream consumption: the
fetched batch is zero-copy-sliced into chunk_rows-sized record batches
(RecordBatch::slice shares the underlying Arrow buffers — no row copy), and
each slice is written and dropped before the next, so peak encoder memory
(parquet row-group buffers, CSV/JSON string buffers, etc.) is bounded by
chunk_rows rather than the whole result, and each output file is bounded
too. Reading back with read_chunk per file then streams chunk-by-chunk.
When the upstream is the lakehouse rather than FalkorDB, the Iceberg export
path ([export_cypher_to_iceberg], feature skade) already appends in
batches; this function is the GraphAr-file equivalent.
chunk_rows must be non-zero; 0 is treated as 1. An empty result still
writes a single empty chunk0.<ext> so consumers always find a chunk file.
use graphar_flight::{FalkorExecutor, export::export_cypher_chunked};
use graphar::FileType;
let mut exec = FalkorExecutor::connect("redis://127.0.0.1:6379", "social").await?;
let report = export_cypher_chunked(
&mut exec,
"MATCH (n:Person) RETURN n._gar_id AS id, n.name AS name",
"/tmp/people_chunks",
FileType::Parquet,
100_000, // ≤ 100k rows per chunk file
).await?;
println!("exported {} rows across {} chunks", report.rows, report.chunks.len());