Struct polars::prelude::LazyFrame

pub struct LazyFrame {
    pub logical_plan: LogicalPlan,
    /* private fields */
}

Available on crate feature lazy only.

Lazy abstraction over an eager DataFrame. It really is an abstraction over a logical plan. The methods of this struct will incrementally modify a logical plan until output is requested (via collect).

Fields

logical_plan: LogicalPlan

Implementations

impl LazyFrame

pub fn to_dot(&self, optimized: bool) -> Result<String, PolarsError>

Get a dot language representation of the LogicalPlan.

impl LazyFrame

pub fn schema(&self) -> Result<Arc<Schema>, PolarsError>

Get a handle to the schema — a map from column names to data types — of the current LazyFrame computation.

Returns an Err if the logical plan has already encountered an error (i.e., if self.collect() would fail), Ok otherwise.

pub fn get_current_optimizations(&self) -> OptState

Get current optimizations.

pub fn with_optimizations(self, opt_state: OptState) -> LazyFrame

Set allowed optimizations.

pub fn without_optimizations(self) -> LazyFrame

Turn off all optimizations.

pub fn with_projection_pushdown(self, toggle: bool) -> LazyFrame

Toggle projection pushdown optimization.

pub fn with_predicate_pushdown(self, toggle: bool) -> LazyFrame

Toggle predicate pushdown optimization.

pub fn with_type_coercion(self, toggle: bool) -> LazyFrame

Toggle type coercion optimization.

pub fn with_simplify_expr(self, toggle: bool) -> LazyFrame

Toggle expression simplification optimization on or off.

pub fn with_slice_pushdown(self, toggle: bool) -> LazyFrame

Toggle slice pushdown optimization.

pub fn with_streaming(self, toggle: bool) -> LazyFrame

Allow (partial) streaming engine.

pub fn _with_eager(self, toggle: bool) -> LazyFrame

pub fn describe_plan(&self) -> String

Return a String describing the naive (un-optimized) logical plan.

pub fn describe_optimized_plan(&self) -> Result<String, PolarsError>

Return a String describing the optimized logical plan.

Returns Err if optimizing the logical plan fails.

pub fn explain(&self, optimized: bool) -> Result<String, PolarsError>

Return a String describing the logical plan.

If optimized is true, explains the optimized plan. If optimized is `false, explains the naive, un-optimized plan.

pub fn sort(self, by_column: &str, options: SortOptions) -> LazyFrame

Add a sort operation to the logical plan.

Sorts the LazyFrame by the column name specified using the provided options.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

/// Sort DataFrame by 'sepal.width' column
fn example(df: DataFrame) -> LazyFrame {
      df.lazy()
        .sort("sepal.width", Default::default())
}

pub fn sort_by_exprs<E, B>( self, by_exprs: E, descending: B, nulls_last: bool, maintain_order: bool ) -> LazyFramewhere E: AsRef<[Expr]>, B: AsRef<[bool]>,

Add a sort operation to the logical plan.

Sorts the LazyFrame by the provided list of expressions, which will be turned into concrete columns before sorting. reverse is a list of bool, the same length as by_exprs, that specifies whether each corresponding expression will be sorted ascending (false) or descending (true).

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

/// Sort DataFrame by 'sepal.width' column
fn example(df: DataFrame) -> LazyFrame {
      df.lazy()
        .sort_by_exprs(vec![col("sepal.width")], vec![false], false, false)
}

pub fn top_k<E, B>( self, k: u32, by_exprs: E, descending: B, nulls_last: bool, maintain_order: bool ) -> LazyFramewhere E: AsRef<[Expr]>, B: AsRef<[bool]>,

pub fn bottom_k<E, B>( self, k: u32, by_exprs: E, descending: B, nulls_last: bool, maintain_order: bool ) -> LazyFramewhere E: AsRef<[Expr]>, B: AsRef<[bool]>,

pub fn reverse(self) -> LazyFrame

Reverse the DataFrame from top to bottom.

Row i becomes row number_of_rows - i - 1.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

fn example(df: DataFrame) -> LazyFrame {
      df.lazy()
        .reverse()
}

pub fn rename<I, J, T, S>(self, existing: I, new: J) -> LazyFramewhere I: IntoIterator<Item = T>, J: IntoIterator<Item = S>, T: AsRef<str>, S: AsRef<str>,

Rename columns in the DataFrame.

existing and new are iterables of the same length containing the old and corresponding new column names. Renaming happens to all existing columns simultaneously, not iteratively. (In particular, all columns in existing must already exist in the LazyFrame when rename is called.)

pub fn drop_columns<I, T>(self, columns: I) -> LazyFramewhere I: IntoIterator<Item = T>, T: AsRef<str>,

Removes columns from the DataFrame. Note that it’s better to only select the columns you need and let the projection pushdown optimize away the unneeded columns.

pub fn shift<E>(self, n: E) -> LazyFramewhere E: Into<Expr>,

Shift the values by a given period and fill the parts that will be empty due to this operation with Nones.

See the method on Series for more info on the shift operation.

pub fn shift_and_fill<E>(self, n: E, fill_value: E) -> LazyFramewhere E: Into<Expr>,

Shift the values by a given period and fill the parts that will be empty due to this operation with the result of the fill_value expression.

See the method on Series for more info on the shift operation.

pub fn fill_null<E>(self, fill_value: E) -> LazyFramewhere E: Into<Expr>,

Fill None values in the DataFrame with an expression.

pub fn fill_nan<E>(self, fill_value: E) -> LazyFramewhere E: Into<Expr>,

Fill NaN values in the DataFrame with an expression.

pub fn cache(self) -> LazyFrame

Caches the result into a new LazyFrame.

This should be used to prevent computations running multiple times.

pub fn cast( self, dtypes: HashMap<&str, DataType, RandomState>, strict: bool ) -> LazyFrame

Cast named frame columns, resulting in a new LazyFrame with updated dtypes

pub fn cast_all(self, dtype: DataType, strict: bool) -> LazyFrame

Cast all frame columns to the given dtype, resulting in a new LazyFrame

pub fn fetch(self, n_rows: usize) -> Result<DataFrame, PolarsError>

Fetch is like a collect operation, but it overwrites the number of rows read by every scan operation. This is a utility that helps debug a query on a smaller number of rows.

Note that the fetch does not guarantee the final number of rows in the DataFrame. Filter, join operations and a lower number of rows available in the scanned file influence the final number of rows.

pub fn optimize( self, lp_arena: &mut Arena<ALogicalPlan>, expr_arena: &mut Arena<AExpr> ) -> Result<Node, PolarsError>

pub fn to_alp_optimized( self ) -> Result<(Node, Arena<ALogicalPlan>, Arena<AExpr>), PolarsError>

pub fn to_alp( self ) -> Result<(Node, Arena<ALogicalPlan>, Arena<AExpr>), PolarsError>

pub fn collect(self) -> Result<DataFrame, PolarsError>

Execute all the lazy operations and collect them into a DataFrame.

The query is optimized prior to execution.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

fn example(df: DataFrame) -> PolarsResult<DataFrame> {
    df.lazy()
      .group_by([col("foo")])
      .agg([col("bar").sum(), col("ham").mean().alias("avg_ham")])
      .collect()
}

pub fn profile(self) -> Result<(DataFrame, DataFrame), PolarsError>

Profile a LazyFrame.

This will run the query and return a tuple containing the materialized DataFrame and a DataFrame that contains profiling information of each node that is executed.

The units of the timings are microseconds.

pub fn sink_parquet( self, path: PathBuf, options: ParquetWriteOptions ) -> Result<(), PolarsError>

Available on crate feature parquet only.

Stream a query result into a parquet file. This is useful if the final result doesn’t fit into memory. This methods will return an error if the query cannot be completely done in a streaming fashion.

pub fn sink_ipc( self, path: PathBuf, options: IpcWriterOptions ) -> Result<(), PolarsError>

Available on crate feature ipc only.

Stream a query result into an ipc/arrow file. This is useful if the final result doesn’t fit into memory. This methods will return an error if the query cannot be completely done in a streaming fashion.

pub fn sink_csv( self, path: PathBuf, options: CsvWriterOptions ) -> Result<(), PolarsError>

Available on crate feature csv only.

Stream a query result into an csv file. This is useful if the final result doesn’t fit into memory. This methods will return an error if the query cannot be completely done in a streaming fashion.

pub fn filter(self, predicate: Expr) -> LazyFrame

Filter by some predicate expression.

The expression must yield boolean values.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

fn example(df: DataFrame) -> LazyFrame {
      df.lazy()
        .filter(col("sepal.width").is_not_null())
        .select(&[col("sepal.width"), col("sepal.length")])
}

pub fn select<E>(self, exprs: E) -> LazyFramewhere E: AsRef<[Expr]>,

Select (and optionally rename, with alias) columns from the query.

Columns can be selected with col; If you want to select all columns use col("*").

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

/// This function selects column "foo" and column "bar".
/// Column "bar" is renamed to "ham".
fn example(df: DataFrame) -> LazyFrame {
      df.lazy()
        .select(&[col("foo"),
                  col("bar").alias("ham")])
}

/// This function selects all columns except "foo"
fn exclude_a_column(df: DataFrame) -> LazyFrame {
      df.lazy()
        .select(&[col("*").exclude(["foo"])])
}

pub fn select_seq<E>(self, exprs: E) -> LazyFramewhere E: AsRef<[Expr]>,

pub fn group_by<E, IE>(self, by: E) -> LazyGroupBywhere E: AsRef<[IE]>, IE: Into<Expr> + Clone,

Performs a “group-by” on a LazyFrame, producing a LazyGroupBy, which can subsequently be aggregated.

Takes a list of expressions to group on.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;
use arrow::legacy::prelude::QuantileInterpolOptions;

fn example(df: DataFrame) -> LazyFrame {
      df.lazy()
       .group_by([col("date")])
       .agg([
           col("rain").min().alias("min_rain"),
           col("rain").sum().alias("sum_rain"),
           col("rain").quantile(lit(0.5), QuantileInterpolOptions::Nearest).alias("median_rain"),
       ])
}

pub fn group_by_rolling<E>( self, index_column: Expr, by: E, options: RollingGroupOptions ) -> LazyGroupBywhere E: AsRef<[Expr]>,

Available on crate feature dynamic_group_by only.

Create rolling groups based on a time column.

Also works for index values of type Int32 or Int64.

Different from a group_by_dynamic, the windows are now determined by the individual values and are not of constant intervals. For constant intervals use group_by_dynamic

pub fn group_by_dynamic<E>( self, index_column: Expr, by: E, options: DynamicGroupOptions ) -> LazyGroupBywhere E: AsRef<[Expr]>,

Available on crate feature dynamic_group_by only.

Group based on a time value (or index value of type Int32, Int64).

Time windows are calculated and rows are assigned to windows. Different from a normal group_by is that a row can be member of multiple groups. The time/index window could be seen as a rolling window, with a window size determined by dates/times/values instead of slots in the DataFrame.

A window is defined by:

• every: interval of the window
• period: length of the window
• offset: offset of the window

The by argument should be empty [] if you don’t want to combine this with a ordinary group_by on these keys.

pub fn group_by_stable<E, IE>(self, by: E) -> LazyGroupBywhere E: AsRef<[IE]>, IE: Into<Expr> + Clone,

Similar to group_by, but order of the DataFrame is maintained.

pub fn cross_join(self, other: LazyFrame) -> LazyFrame

Available on crate feature cross_join only.

Creates the cartesian product from both frames, preserving the order of the left keys.

pub fn left_join<E>( self, other: LazyFrame, left_on: E, right_on: E ) -> LazyFramewhere E: Into<Expr>,

Left join this query with another lazy query.

Matches on the values of the expressions left_on and right_on. For more flexible join logic, see join or join_builder.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;
fn left_join_dataframes(ldf: LazyFrame, other: LazyFrame) -> LazyFrame {
        ldf
        .left_join(other, col("foo"), col("bar"))
}

pub fn inner_join<E>( self, other: LazyFrame, left_on: E, right_on: E ) -> LazyFramewhere E: Into<Expr>,

Inner join this query with another lazy query.

Matches on the values of the expressions left_on and right_on. For more flexible join logic, see join or join_builder.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;
fn inner_join_dataframes(ldf: LazyFrame, other: LazyFrame) -> LazyFrame {
        ldf
        .inner_join(other, col("foo"), col("bar").cast(DataType::Utf8))
}

pub fn outer_join<E>( self, other: LazyFrame, left_on: E, right_on: E ) -> LazyFramewhere E: Into<Expr>,

Outer join this query with another lazy query.

Matches on the values of the expressions left_on and right_on. For more flexible join logic, see join or join_builder.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;
fn outer_join_dataframes(ldf: LazyFrame, other: LazyFrame) -> LazyFrame {
        ldf
        .outer_join(other, col("foo"), col("bar"))
}

pub fn join<E>( self, other: LazyFrame, left_on: E, right_on: E, args: JoinArgs ) -> LazyFramewhere E: AsRef<[Expr]>,

Generic function to join two LazyFrames.

join can join on multiple columns, given as two list of expressions, and with a JoinType specified by how. Non-joined column names in the right DataFrame that already exist in this DataFrame are suffixed with "_right". For control over how columns are renamed and parallelization options, use join_builder.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;

fn example(ldf: LazyFrame, other: LazyFrame) -> LazyFrame {
        ldf
        .join(other, [col("foo"), col("bar")], [col("foo"), col("bar")], JoinArgs::new(JoinType::Inner))
}

pub fn join_builder(self) -> JoinBuilder

Consume self and return a JoinBuilder to customize a join on this LazyFrame.

After the JoinBuilder has been created and set up, calling finish() on it will give back the LazyFrame representing the join operation.

pub fn with_column(self, expr: Expr) -> LazyFrame

Add or replace a column, given as an expression, to a DataFrame.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;
fn add_column(df: DataFrame) -> LazyFrame {
    df.lazy()
        .with_column(
            when(col("sepal.length").lt(lit(5.0)))
            .then(lit(10))
            .otherwise(lit(1))
            .alias("new_column_name"),
        )
}

pub fn with_columns<E>(self, exprs: E) -> LazyFramewhere E: AsRef<[Expr]>,

Add or replace multiple columns, given as expressions, to a DataFrame.

Example

use polars_core::prelude::*;
use polars_lazy::prelude::*;
fn add_columns(df: DataFrame) -> LazyFrame {
    df.lazy()
        .with_columns(
            vec![lit(10).alias("foo"), lit(100).alias("bar")]
         )
}

pub fn with_columns_seq<E>(self, exprs: E) -> LazyFramewhere E: AsRef<[Expr]>,

Add or replace multiple columns to a DataFrame, but evaluate them sequentially.

pub fn with_context<C>(self, contexts: C) -> LazyFramewhere C: AsRef<[LazyFrame]>,

pub fn max(self) -> LazyFrame

Aggregate all the columns as their maximum values.

Aggregated columns will have the same names as the original columns.

pub fn min(self) -> LazyFrame

Aggregate all the columns as their minimum values.

Aggregated columns will have the same names as the original columns.

pub fn sum(self) -> LazyFrame

Aggregate all the columns as their sum values.

Aggregated columns will have the same names as the original columns.

• Boolean columns will sum to a u32 containing the number of trues.
• For integer columns, the ordinary checks for overflow are performed: if running in debug mode, overflows will panic, whereas in release mode overflows will silently wrap.
• String columns will sum to None.

pub fn mean(self) -> LazyFrame

Aggregate all the columns as their mean values.

• Boolean and integer columns are converted to f64 before computing the mean.
• String columns will have a mean of None.

pub fn median(self) -> LazyFrame

Aggregate all the columns as their median values.

• Boolean and integer results are converted to f64. However, they are still susceptible to overflow before this conversion occurs.
• String columns will sum to None.

pub fn quantile( self, quantile: Expr, interpol: QuantileInterpolOptions ) -> LazyFrame

Aggregate all the columns as their quantile values.

pub fn std(self, ddof: u8) -> LazyFrame

Aggregate all the columns as their standard deviation values.

ddof is the “Delta Degrees of Freedom”; N - ddof will be the denominator when computing the variance, where N is the number of rows.

In standard statistical practice, ddof=1 provides an unbiased estimator of the variance of a hypothetical infinite population. ddof=0 provides a maximum likelihood estimate of the variance for normally distributed variables. The standard deviation computed in this function is the square root of the estimated variance, so even with ddof=1, it will not be an unbiased estimate of the standard deviation per se.

Source: Numpy

pub fn var(self, ddof: u8) -> LazyFrame

Aggregate all the columns as their variance values.

ddof is the “Delta Degrees of Freedom”; N - ddof will be the denominator when computing the variance, where N is the number of rows.

In standard statistical practice, ddof=1 provides an unbiased estimator of the variance of a hypothetical infinite population. ddof=0 provides a maximum likelihood estimate of the variance for normally distributed variables.

Source: Numpy

pub fn explode<E, IE>(self, columns: E) -> LazyFramewhere E: AsRef<[IE]>, IE: Into<Expr> + Clone,

Apply explode operation. See eager explode.

pub fn null_count(self) -> LazyFrame

Aggregate all the columns as the sum of their null value count.

pub fn unique_stable( self, subset: Option<Vec<String>>, keep_strategy: UniqueKeepStrategy ) -> LazyFrame

Drop non-unique rows and maintain the order of kept rows.

subset is an optional Vec of column names to consider for uniqueness; if None, all columns are considered.

pub fn unique( self, subset: Option<Vec<String>>, keep_strategy: UniqueKeepStrategy ) -> LazyFrame

Drop non-unique rows without maintaining the order of kept rows.

The order of the kept rows may change; to maintain the original row order, use unique_stable.

subset is an optional Vec of column names to consider for uniqueness; if None, all columns are considered.

pub fn drop_nulls(self, subset: Option<Vec<Expr>>) -> LazyFrame

Drop rows containing None.

subset is an optional Vec of column names to consider for nulls; if None, all columns are considered.

pub fn slice(self, offset: i64, len: u32) -> LazyFrame

Slice the DataFrame using an offset (starting row) and a length.

If offset is negative, it is counted from the end of the DataFrame. For instance, lf.slice(-5, 3) gets three rows, starting at the row fifth from the end.

If offset and len are such that the slice extends beyond the end of the DataFrame, the portion between offset and the end will be returned. In this case, the number of rows in the returned DataFrame will be less than len.

pub fn first(self) -> LazyFrame

Get the first row.

Equivalent to self.slice(0, 1).

pub fn last(self) -> LazyFrame

Get the last row.

Equivalent to self.slice(-1, 1).

pub fn tail(self, n: u32) -> LazyFrame

Get the last n rows.

Equivalent to self.slice(-(n as i64), n).

pub fn melt(self, args: MeltArgs) -> LazyFrame

Melt the DataFrame from wide to long format.

See MeltArgs for information on how to melt a DataFrame.

pub fn limit(self, n: u32) -> LazyFrame

Limit the DataFrame to the first n rows.

Note if you don’t want the rows to be scanned, use fetch.

pub fn map<F>( self, function: F, optimizations: OptState, schema: Option<Arc<dyn UdfSchema>>, name: Option<&'static str> ) -> LazyFramewhere F: 'static + Fn(DataFrame) -> Result<DataFrame, PolarsError> + Send + Sync,

Apply a function/closure once the logical plan get executed.

The function has access to the whole materialized DataFrame at the time it is called.

To apply specific functions to specific columns, use Expr::map in conjunction with LazyFrame::with_column or with_columns.

Warning

This can blow up in your face if the schema is changed due to the operation. The optimizer relies on a correct schema.

You can toggle certain optimizations off.

pub fn with_row_count(self, name: &str, offset: Option<u32>) -> LazyFrame

Add a new column at index 0 that counts the rows.

name is the name of the new column. offset is where to start counting from; if None, it is set to 0.

Warning

This can have a negative effect on query performance. This may for instance block predicate pushdown optimization.

pub fn unnest<I, S>(self, cols: I) -> LazyFramewhere I: IntoIterator<Item = S>, S: AsRef<str>,

Available on crate feature dtype-struct only.

Unnest the given Struct columns: the fields of the Struct type will be inserted as columns.

impl LazyFrame

pub fn anonymous_scan( function: Arc<dyn AnonymousScan>, args: ScanArgsAnonymous ) -> Result<LazyFrame, PolarsError>

impl LazyFrame

pub fn scan_ipc( path: impl AsRef<Path>, args: ScanArgsIpc ) -> Result<LazyFrame, PolarsError>

Create a LazyFrame directly from a ipc scan.

pub fn scan_ipc_files( paths: Arc<[PathBuf]>, args: ScanArgsIpc ) -> Result<LazyFrame, PolarsError>

impl LazyFrame

pub fn scan_parquet( path: impl AsRef<Path>, args: ScanArgsParquet ) -> Result<LazyFrame, PolarsError>

Create a LazyFrame directly from a parquet scan.

pub fn scan_parquet_files( paths: Arc<[PathBuf]>, args: ScanArgsParquet ) -> Result<LazyFrame, PolarsError>

Create a LazyFrame directly from a parquet scan.

Trait Implementations

impl Clone for LazyFrame

fn clone(&self) -> LazyFrame

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for LazyFrame

fn default() -> LazyFrame

Returns the “default value” for a type.

impl From<LogicalPlan> for LazyFrame

fn from(plan: LogicalPlan) -> LazyFrame

Converts to this type from the input type.