buoyant_kernel 0.25.0

Buoyant Data distribution of delta-kernel
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

//! Plan containers ([`Plan`], [`PlanNode`]).

pub use super::nodes::Operator;

// ============================================================================
// Plan nodes
// ============================================================================

/// One node in a plan: an [`Operator`] and the indices of its input nodes.
///
/// A node is identified by its position in [`Plan::nodes`]; `inputs` lists those indices for
/// the upstream nodes this operator reads from. `inputs` order is interpreted per [`Operator`]
/// (e.g. for `Operator::SemiJoin` the convention is `[probe, build]`). `Operator::UnionAll`
/// emits the rows of all inputs regardless of input order.
#[derive(Debug, Clone)]
pub struct PlanNode {
    pub op: Operator,
    pub inputs: Vec<usize>,
}

impl PlanNode {
    /// A node applying `op` over the nodes at `inputs` (indices into [`Plan::nodes`]).
    pub fn new(op: impl Into<Operator>, inputs: Vec<usize>) -> Self {
        Self {
            op: op.into(),
            inputs,
        }
    }
}

// ============================================================================
// Plans
// ============================================================================

/// A plan: an ordered sequence of [`PlanNode`]s forming a dataflow DAG.
///
/// A node is identified by its index in `nodes`. Each [`PlanNode`] pairs an operator with the
/// indices of its inputs:
///
/// - `op` ([`Operator`]) is the operator: a source like `ScanParquet` or a transform like
///   `Project`.
/// - `inputs` is a `Vec<usize>` naming the indices of the upstream nodes the operator reads from.
///
/// A node depends on another when one of its `inputs` is that node's index. `nodes` is stored in
/// topological order: every node appears after the nodes it consumes (each input index is
/// strictly less than the node's own index), so an engine can evaluate `nodes` in slice order;
/// each node's inputs are guaranteed bound by the time the node is reached.
///
/// A well-formed `Plan` has at least one node. The **terminal node** is always the last entry in
/// `nodes`: no other node lists its index in `inputs`, and its rows are the value the engine
/// streams to the caller.
///
/// # Optimization
///
/// For the best performance, connectors are encouraged to run kernel-produced
/// plans through their query optimizer before execution (e.g. to fold adjacent
/// filters, merge scans over the same files, or choose physical join and scan
/// strategies).
///
/// # Example
///
/// A five-node plan: two independent scans, each filtered, then unioned. The `nodes`
/// `Vec<PlanNode>`:
///
/// ```text
/// Plan {
///     nodes: vec![
///         PlanNode { op: ScanParquet(..), inputs: vec![]     },  // node 0
///         PlanNode { op: ScanParquet(..), inputs: vec![]     },  // node 1
///         PlanNode { op: Filter(..),      inputs: vec![0]    },  // node 2
///         PlanNode { op: Filter(..),      inputs: vec![1]    },  // node 3
///         PlanNode { op: UnionAll(..),    inputs: vec![2, 3] },  // node 4
///     ],
/// }
/// ```
///
/// The dataflow DAG this encodes:
///
/// ```text
///    ScanParquet [0]     ScanParquet [1]
///           |                    |
///           v                    v
///       Filter [2]           Filter [3]
///           |                    |
///           +---------+----------+
///                     v
///               UnionAll [4]   <-- terminal (last node)
/// ```
///
/// The engine evaluates the nodes in the order of the `nodes` vector: nodes `0` and `1` first
/// (sources), then `2` and `3`, then `4`. The engine streams the rows produced at the terminal
/// node to the caller.
#[derive(Debug, Clone)]
pub struct Plan {
    pub nodes: Vec<PlanNode>,
}