vec_parallel

A library for building vectors in parallel using async tasks.

This crate provides an efficient, executor-agnostic way to construct Vec<T> by dividing the work into multiple async tasks that can run concurrently. It's particularly useful for CPU-bound initialization tasks where elements can be computed independently.

Overview

vec_parallel allows you to parallelize the construction of vectors by splitting the work across multiple async tasks. Each task is responsible for computing a portion of the vector, writing directly to the final memory location to avoid unnecessary copies.

Key Features

Flexible parallelization strategies: Control task creation with [Strategy]
Zero-copy construction: Elements are written directly to their final location
Executor-agnostic: Works with any async runtime (tokio, async-std, smol, etc.)
Optional executor integration: Use the some_executor feature for convenient spawning
WASM support: Works in browser environments with wasm-bindgen
Safe abstraction: Careful use of unsafe code with documented invariants

Usage Patterns

Basic Usage

use vec_parallel::{build_vec, Strategy};

// Build a vector of squares using multiple tasks
let builder = build_vec(100, Strategy::TasksPerCore(4), |i| i * i);

// Run the tasks (in a real application, these would be spawned on an executor)
for task in builder.tasks {
    test_executors::spin_on(task);
}

// Get the final result
let squares = test_executors::spin_on(builder.result);
assert_eq!(squares[10], 100); // 10² = 100

With Async Executors

use vec_parallel::{build_vec, Strategy};

// With tokio (or any async runtime)
let builder = build_vec(1000, Strategy::TasksPerCore(4), |i| {
    // Expensive computation
    (0..100).map(|j| (i + j) * 2).sum::<usize>()
});

// Spawn tasks on your executor
for task in builder.tasks {
    // In a real app: tokio::spawn(task);
    test_executors::spin_on(task);
}

// Await the result
let result = test_executors::spin_on(builder.result);
assert_eq!(result.len(), 1000);

Choosing a Strategy

The [Strategy] enum controls how work is divided:

[Strategy::One]: No parallelism, single task
[Strategy::Tasks(n)]: Exactly n tasks
[Strategy::Max]: One task per element (maximum parallelism)
[Strategy::TasksPerCore(n)]: n tasks per CPU core (recommended for CPU-bound work)

Performance Considerations

For CPU-bound work, use [Strategy::TasksPerCore(4)] to [Strategy::TasksPerCore(8)]
For I/O-bound work, consider higher task counts
For small vectors (<100 elements), parallelization overhead may not be worth it
The library uses atomic operations for synchronization, avoiding locks

Safety

This library uses unsafe code internally for performance, but maintains safety through:

Non-overlapping slice assignments for each task
Atomic counters for task completion tracking
Careful lifetime management with Arc and Weak references
All unsafe operations are documented with their safety invariants

Optional Features

some_executor: Enables integration with the some_executor crate for convenient task spawning

vec_parallel 0.1.3