Struct WorkQueue

pub struct WorkQueue<T> { /* private fields */ }

Multi-producer, multi-consumer async job queue with end conditions.

Keeps track of in-progress jobs and can indicate to workers that all work has been finished. Intended to be used for jobs that will spawn other jobs during processing, but stop once all jobs have finished.

Example use

#![allow(non_snake_case)]
type DemoJob = (&'static str, usize);

/// Job processing function.
///
/// For a given job `(name, n)`, returns two jobs with `n = n - 1`, unless `n == 0`.
async fn process_job(job: DemoJob) -> Vec<DemoJob> {
    tokio::time::sleep(Duration::from_millis(25)).await;

    let (tag, n) = job;

    if n == 0 {
        Vec::new()
    } else {
        vec![(tag, n - 1), (tag, n - 1)]
    }
}

/// Job-processing worker.
///
/// `id` is the worker ID for logging.
async fn worker(id: usize, q: Arc<WorkQueue<DemoJob>>) {
    println!("worker {}: init", id);

    while let Some(job) = q.next_job().await {
        println!("worker {}: start job {:?}", id, job.inner());
        for new_job in process_job(job.inner().clone()).await {
            q.push_job(new_job);
        }
        println!("worker {}: finish job {:?}", id, job.inner());
    }

    println!("worker {}: shutting down", id);
}

const WORKER_COUNT: usize = 3;
let q = Arc::new(WorkQueue::default());
q.push_job(("A", 3));

let workers: FuturesUnordered<_> = (0..WORKER_COUNT).map(|id| worker(id, q.clone())).collect();

// Wait for all workers to finish.
workers.for_each(|_| async move {}).await;

Implementations

impl<T> WorkQueue<T>

pub async fn next_job(self: &Arc<Self>) -> Option<JobHandle<T>>

Pop a job from the queue.

If there is a job in the queue, returns the job and increases the internal in progress counter by one.

If there are still jobs in progress, but none queued, waits until either of these conditions changes, then retries.

If there are no jobs available and no jobs in progress, returns None.

pub fn push_job(&self, job: T)

Pushes a job onto the queue.

If there are any worker waiting on next_job, one of them will receive the job.

pub fn num_jobs(&self) -> usize

Returns the number of jobs in the queue.

pub fn to_stream(self: Arc<Self>) -> impl Stream<Item = JobHandle<T>>

Creates a streaming consumer of the work queue.

Trait Implementations

impl<T: Debug> Debug for WorkQueue<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T> Default for WorkQueue<T>

fn default() -> Self

Returns the “default value” for a type.