Skip to main content

Module df

Module df 

Source
Expand description

std.df — Polars-backed query ops over arrow.Table (#427).

The companion to std.arrow. Where std.arrow covers construction + column reductions, std.df covers the query-shaped operations — filter, sort, group_by + agg, join — that Polars already does vectorised + parallel. Same input/output type (Value::ArrowTable); the Polars DataFrame is internal plumbing.

Conversion across the arrow-rs ↔ polars-arrow boundary is a column-by-column copy (typed buffer → Vec<T>Series). For primitive columns this is a memcpy-speed walk; for String columns it copies the offsets + bytes. On the scale lex-frame cares about (≤ 10M rows) this is ~10 ms each direction, negligible compared to the savings on the actual query.

Functions§

dispatch
read_csv_at_polars
Read a CSV through Polars’ parallel reader and hand back the usual Value::ArrowTable. This is what arrow.read_csv dispatches to when the df feature is on (the default for the lex toolchain): the arrow-rs CSV reader is single-threaded and was measured at ~10x the wall time of Polars’ on a 1M-row file — it dominated every read-then-query pipeline (see lex-frame’s bench/REPORT.md).