Struct StaticExecutionGraph 

pub struct StaticExecutionGraph { /* private fields */ }

ExecutionGraph implementation which generates all stages on job submission time

Implementations

impl StaticExecutionGraph

pub fn new( scheduler_id: &str, job_id: &JobId, job_name: &str, session_id: &str, plan: Arc<dyn ExecutionPlan>, queued_at: u64, session_config: Arc<SessionConfig>, planner: &mut dyn DistributedPlanner, logical_plan: Option<String>, ) -> Result<Self>

Creates a new ExecutionGraph from a physical execution plan.

This will use the DistributedPlanner to break the plan into stages and build the DAG structure needed for distributed execution.

Trait Implementations

impl Clone for StaticExecutionGraph

fn clone(&self) -> StaticExecutionGraph

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StaticExecutionGraph

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ExecutionGraph for StaticExecutionGraph

fn is_successful(&self) -> bool

An ExecutionGraph is successful if all its stages are successful

fn revive(&mut self) -> bool

Revive the execution graph by converting the resolved stages to running stages If any stages are converted, return true; else false.

fn update_task_status( &mut self, executor: &ExecutorMetadata, task_statuses: Vec<TaskStatus>, max_task_failures: usize, max_stage_failures: usize, ) -> Result<Vec<QueryStageSchedulerEvent>>

Update task statuses and task metrics in the graph. This will also push shuffle partitions to their respective shuffle read stages.

fn running_stages(&self) -> Vec<usize>

Return all the currently running stage ids

fn running_tasks(&self) -> Vec<RunningTaskInfo>

Return all currently running tasks along with the executor ID on which they are assigned

fn available_tasks(&self) -> usize

Total number of tasks in this plan that are ready for scheduling

fn reset_stages_on_lost_executor( &mut self, executor_id: &str, ) -> Result<(HashSet<usize>, Vec<RunningTaskInfo>)>

Reset running and successful stages on a given executor This will first check the unresolved/resolved/running stages and reset the running tasks and successful tasks. Then it will check the successful stage and whether there are running parent stages need to read shuffle from it. If yes, reset the successful tasks and roll back the resolved shuffle recursively.

Returns the reset stage ids and running tasks should be killed

fn resolve_stage(&mut self, stage_id: usize) -> Result<bool>

Convert unresolved stage to be resolved

fn succeed_stage(&mut self, stage_id: usize) -> bool

Convert running stage to be successful

fn fail_stage(&mut self, stage_id: usize, err_msg: String) -> bool

Convert running stage to be failed

fn rollback_running_stage( &mut self, stage_id: usize, failure_reasons: HashSet<String>, ) -> Result<Vec<RunningTaskInfo>>

Convert running stage to be unresolved, Returns a Vec of RunningTaskInfo for running tasks in this stage.

fn rollback_resolved_stage(&mut self, stage_id: usize) -> Result<bool>

Convert resolved stage to be unresolved

fn rerun_successful_stage(&mut self, stage_id: usize) -> bool

Convert successful stage to be running

fn fail_job(&mut self, error: String)

fail job with error message

fn succeed_job(&mut self) -> Result<()>

Mark the job success

fn cloned(&self) -> ExecutionGraphBox

Clones execution graph

fn job_id(&self) -> &JobId

Returns the job ID for this execution graph.

fn job_name(&self) -> &str

Returns the job name for this execution graph.

fn session_id(&self) -> &str

Returns the session ID associated with this job.

fn session_config(&self) -> Arc<SessionConfig>

Returns the session config associated with this job.

fn status(&self) -> &JobStatus

Returns the current status of the job.

fn logical_plan(&self) -> Option<&str>

Returns the logical plan as a string, if captured at submission time.

fn physical_plan(&self) -> Arc<dyn ExecutionPlan>

Returns the physical plan as a string, if captured at submission time.

fn start_time(&self) -> u64

Returns the timestamp when this job started execution.

fn end_time(&self) -> u64

Returns the timestamp when this job started execution.

fn completed_stages(&self) -> usize

Number of completed stages

fn fetch_running_stage( &mut self, black_list: &[usize], ) -> Option<(&mut RunningStage, &mut usize)>

Fetches a running stage that has available tasks, excluding stages in the blacklist.

fn update_status(&mut self, status: JobStatus)

Updates the job status.

fn output_locations(&self) -> Vec<PartitionLocation>

Returns the output partition locations for the final stage results.

fn stages(&self) -> &HashMap<usize, ExecutionStage>

Exposes executions stages and stage id’s

fn stage_count(&self) -> usize

Returns the total number of stages in this execution graph.

fn abort_running(&mut self, error: String) -> Vec<RunningTaskInfo>

Abort a running job: fail it, transition every running stage to Failed, and return the in-flight tasks that should be cancelled. Used for both the failure and cancellation teardown paths.

fn intermediate_stage_ids(&self) -> Vec<u32>

Stage ids of all non-final (intermediate) stages — those whose output_links is non-empty. The final stage(s) are excluded.