Skip to main content

export_cypher_chunked

Function export_cypher_chunked 

Source
pub async fn export_cypher_chunked(
    executor: &mut FalkorExecutor,
    cypher: &str,
    dir: impl AsRef<Path>,
    file_type: FileType,
    chunk_rows: usize,
) -> Result<ExportReport>
Expand description

Run cypher and write the result as multiple GraphAr chunk files — chunk0.<ext>, chunk1.<ext>, … — under dir, each holding at most chunk_rows rows. This is the chunked export for huge results: instead of one monolithic file it lands the result the way GraphAr naturally tiles a vertex/edge collection (one file per row-range), so a VertexReader (or a plain read_chunk per file) can stream the chunks back without ever holding the whole result in memory.

Returns an ExportReport whose chunks lists the files written in order.

§Streaming honesty — why this is slice-and-write, not pull-stream

FalkorDB’s GRAPH.QUERY is a single RESP request/response: the server serializes the entire result set into one reply with no server-side cursor, and FalkorExecutor::query_auto therefore materializes it as one in-memory RecordBatch. There is no API to pull the result in pages, so we cannot bound the read side below the full result. (FalkorDB ≥ 4 exposes GRAPH.QUERY … TIMEOUT/result-set limits but no streaming cursor over RESP.)

What this does bound is the write side and downstream consumption: the fetched batch is zero-copy-sliced into chunk_rows-sized record batches (RecordBatch::slice shares the underlying Arrow buffers — no row copy), and each slice is written and dropped before the next, so peak encoder memory (parquet row-group buffers, CSV/JSON string buffers, etc.) is bounded by chunk_rows rather than the whole result, and each output file is bounded too. Reading back with read_chunk per file then streams chunk-by-chunk.

When the upstream is the lakehouse rather than FalkorDB, the Iceberg export path ([export_cypher_to_iceberg], feature skade) already appends in batches; this function is the GraphAr-file equivalent.

chunk_rows must be non-zero; 0 is treated as 1. An empty result still writes a single empty chunk0.<ext> so consumers always find a chunk file.

use graphar_flight::{FalkorExecutor, export::export_cypher_chunked};
use graphar::FileType;

let mut exec = FalkorExecutor::connect("redis://127.0.0.1:6379", "social").await?;
let report = export_cypher_chunked(
    &mut exec,
    "MATCH (n:Person) RETURN n._gar_id AS id, n.name AS name",
    "/tmp/people_chunks",
    FileType::Parquet,
    100_000, // ≤ 100k rows per chunk file
).await?;
println!("exported {} rows across {} chunks", report.rows, report.chunks.len());