Skip to main content

SHUFFLE_FETCH_BUFFER_ENV

Constant SHUFFLE_FETCH_BUFFER_ENV 

Source
pub const SHUFFLE_FETCH_BUFFER_ENV: &str = "KRISHIV_SHUFFLE_FETCH_BUFFER";
Expand description

Env var overriding how many map fragments ONE reduce partition fetches at once. See the buffered call in ShuffleReadExec::execute.

Deliberately not KRISHIV_SHUFFLE_FETCH_CONCURRENCY: that name is already taken by the executor-wide Flight semaphore (fragment/common.rs, default 8 in-flight requests per process). The two are different limits — this one is per reduce partition, that one is the process-wide ceiling every partition’s fetches must also pass through — and giving them one name would mean a single number silently driving both.