Struct MapReduceSpec

Source

pub struct MapReduceSpec { /* private fields */ }

Expand description

A map-reduce operation.

The mapper is optional: without one, the input is fed straight to the reducer, which is how YTsaurus models a plain shuffle-and-reduce.

Implementations§

Source §

impl MapReduceSpec

Source

pub fn new<I, O, K>( reducer: impl Into<String>, inputs: I, outputs: O, reduce_by: K, ) -> Self
where I: IntoIterator, I::Item: Into<String>, O: IntoIterator, O::Item: Into<String>, K: IntoIterator, K::Item: Into<String>,

A map-reduce running reducer over inputs, grouped by reduce_by.

Source

pub fn with_mapper(self, command: impl Into<String>) -> Self

Adds a mapper phase.

Source

pub fn with_mapper_formats(self, input: DataFormat, output: DataFormat) -> Self

Selects the mapper phase’s input and output data formats.

A Skiff format must contain one schema per corresponding table. It may be called before or after Self::with_mapper.

Source

pub fn with_mapper_skiff_formats( self, input: SkiffFormat, output: SkiffFormat, ) -> Self

Uses validated Skiff formats for the mapper phase.

This compatibility convenience delegates to Self::with_mapper_formats.

Source

pub fn with_reducer_formats(self, input: DataFormat, output: DataFormat) -> Self

Selects the reducer phase’s input and output data formats.

A Skiff format must contain one schema per corresponding table, in the order YTsaurus uses for that phase.

Source

pub fn with_reducer_skiff_formats( self, input: SkiffFormat, output: SkiffFormat, ) -> Self

Uses validated Skiff formats for the reducer phase.

This compatibility convenience delegates to Self::with_reducer_formats.

Source

pub fn with_local_file(self, path: impl Into<String>) -> Self

Adds a Cypress file to both phases.

One binary usually serves both, dispatching on argv[1], so attaching it to each phase separately would only be a way to forget one. Order relative to MapReduceSpec::with_mapper does not matter: files are handed to the phases when the spec is rendered.

Source

pub fn with_local_file_named( self, path: impl Into<String>, name: impl AsRef<str>, ) -> Self

Adds a Cypress file to both phases under a different sandbox name.

See MapSpec::with_local_file_named for why the name matters.

Source

pub fn with_memory_limit(self, bytes: i64) -> Self

Sets the memory limit for both phases, in bytes.

As with the files, order relative to MapReduceSpec::with_mapper does not matter.

Source

pub fn with_sort_by<K>(self, columns: K) -> Self
where K: IntoIterator, K::Item: Into<String>,

Sets the sort columns, when they differ from the reduce columns.

Source

pub fn without_key_switch(self) -> Self

Turns off key_switch delivery to the reducer.

Only useful for a reducer that does not group — with it off, JobReader::groups sees the whole input as one group.

Source

pub fn skiff_table_mismatch(&self) -> Option<String>

Describes a Skiff format that does not match this spec’s table lists.

Only the counts this builder can know. What the mapper writes and what the reducer reads are shuffle streams, and how the output tables are split between the phases depends on mapper_output_table_count, which this builder does not model — a spec that sets it through Self::with_raw therefore has its output side left to the cluster rather than guessed at. The Go SDK declines to check its reduce phase for the same reason.

See MapSpec::skiff_table_mismatch for what an unchecked mismatch costs. Client::start_map_reduce checks this before sending the spec.

Source