Crate datafusion_proto 

Serialize / Deserialize DataFusion Plans to bytes

This crate provides support for serializing and deserializing the following structures to and from bytes:

1. LogicalPlan’s (including Expr),
2. ExecutionPlans (including PhysicalExpr)

Internally, this crate is implemented by converting the plans to protocol buffers using prost.

Version Compatibility

The serialized form are not guaranteed to be compatible across DataFusion versions. A plan serialized with one version of DataFusion may not be able to deserialized with a different version.

See Also

The binary format created by this crate supports the full range of DataFusion plans, but is DataFusion specific. See datafusion-substrait for a crate which can encode many DataFusion plans using the substrait.io standard.

Example: Serializing Exprs

 // Create a new `Expr` a < 32
 let expr = col("a").lt(lit(5i32));

 // Convert it to bytes (for sending over the network, etc.)
 let bytes = expr.to_bytes()?;

 // Decode bytes from somewhere (over network, etc.) back to Expr
 let decoded_expr = Expr::from_bytes(&bytes)?;
 assert_eq!(expr, decoded_expr);

Example: Serializing LogicalPlans

 // Create a plan that scans table 't'
 let ctx = SessionContext::new();
 ctx.register_csv("t1", "tests/testdata/test.csv", CsvReadOptions::default()).await?;
 let plan = ctx.table("t1").await?.into_optimized_plan()?;

 // Convert the plan into bytes (for sending over the network, etc.)
 let bytes = logical_plan_to_bytes(&plan)?;

 // Decode bytes from somewhere (over network, etc.) back to LogicalPlan
 let logical_round_trip = logical_plan_from_bytes(&bytes, &ctx)?;
 assert_eq!(format!("{:?}", plan), format!("{:?}", logical_round_trip));

Example: Serializing ExecutionPlans

 // Create a plan that scans table 't'
 let ctx = SessionContext::new();
 ctx.register_csv("t1", "tests/testdata/test.csv", CsvReadOptions::default()).await?;
 let physical_plan = ctx.table("t1").await?.create_physical_plan().await?;

 // Convert the plan into bytes (for sending over the network, etc.)
 let bytes = physical_plan_to_bytes(physical_plan.clone())?;

 // Decode bytes from somewhere (over network, etc.) back to ExecutionPlan
 let physical_round_trip = physical_plan_from_bytes(&bytes, &ctx)?;
 assert_eq!(format!("{:?}", physical_plan), format!("{:?}", physical_round_trip));

Modules

bytes

Serialization / Deserialization to Bytes

common

generated

logical_plan

physical_plan

protobuf

Macros

convert_box_required

convert_required

into_logical_plan

into_required