Struct GenerationPlan 

pub struct GenerationPlan { /* private fields */ }

A list of generator “parts” (data generator chunks, not TPCH parts) for a single output file.

Controls the parallelization and layout of Parquet files in tpchgen-cli.

Background

A “part” is a logical partition of a particular output table. Each data generator can create parts individually.

For example, the parameters to OrderGenerator::new scale_factor, part_countandpart_counttogether define a partition of theOrder` table.

The entire output table results from generating each of the part_count parts. For example, if part_count is 10, appending parts 1 to 10 results in a complete Order table.

Interesting properties of parts:

1. They are independent of each other, so they can be generated in parallel.
2. They scale. So for example, parts 0..10 with a part_count of 50 will generate the same data as parts 1 with a part_count of 5.

Implication for tpchgen-cli

For tbl and csv files, tpchgen-cli generates num-threads parts in parallel.

For Parquet files, the output file has one row group for each “part”.

Example

use tpchgen_cli::{GenerationPlan, OutputFormat, Table};

let plan = GenerationPlan::try_new(
  Table::Orders,
  OutputFormat::Parquet,
  1.0, // scale factor
  Some(-1), // cli_part
  Some(-1), // cli_parts
   0,
 );
let results = plan.into_iter().collect::<Vec<_>>();
/// assert_eq!(results.len(), 1);

Implementations

impl GenerationPlan

pub fn try_new( table: Table, format: OutputFormat, scale_factor: f64, cli_part: Option<i32>, cli_part_count: Option<i32>, parquet_row_group_bytes: i64, ) -> Result<Self, String>

Returns a GenerationPlan number of parts to generate

Arguments

• cli_part: optional part number to generate (1-based), --part CLI argument
• cli_part_count: optional total number of parts, --parts CLI argument
• parquet_row_group_size: optional parquet row group size, --parquet-row-group-size CLI argument

pub fn partitioned_table(table: Table) -> bool

Return true if the tables is unpartitionable (not parameterized by part count)

pub fn chunk_count(&self) -> usize

Return the number of part(ititions) this plan will generate

Trait Implementations

impl Clone for GenerationPlan

fn clone(&self) -> GenerationPlan

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for GenerationPlan

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for GenerationPlan

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl IntoIterator for GenerationPlan

Converts the GenerationPlan into an iterator of (part_number, num_parts)

type Item = (i32, i32)

The type of the elements being iterated over.

type IntoIter = IntoIter<<GenerationPlan as IntoIterator>::Item>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl PartialEq for GenerationPlan

fn eq(&self, other: &GenerationPlan) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for GenerationPlan