Expand description
std.df — Polars-backed query ops over arrow.Table (#427).
The companion to std.arrow. Where std.arrow covers construction +
column reductions, std.df covers the query-shaped operations —
filter, sort, group_by + agg, join — that Polars already
does vectorised + parallel. Same input/output type (Value::ArrowTable);
the Polars DataFrame is internal plumbing.
Conversion across the arrow-rs ↔ polars-arrow boundary is a
column-by-column copy (typed buffer → Vec<T> → Series). For
primitive columns this is a memcpy-speed walk; for String
columns it copies the offsets + bytes. On the scale lex-frame
cares about (≤ 10M rows) this is ~10 ms each direction, negligible
compared to the savings on the actual query.
Functions§
- dispatch
- read_
csv_ at_ polars - Read a CSV through Polars’ parallel reader and hand back the usual
Value::ArrowTable. This is whatarrow.read_csvdispatches to when thedffeature is on (the default for thelextoolchain): the arrow-rs CSV reader is single-threaded and was measured at ~10x the wall time of Polars’ on a 1M-row file — it dominated every read-then-query pipeline (see lex-frame’s bench/REPORT.md).