cgpu

Reusable CPU/GPU compute execution layer for Rust.

cgpu provides a unified interface for running compute jobs that can execute on the GPU via wgpu or fall back to CPU execution. It handles batch planning, memory management, shader compilation, and telemetry automatically, letting you focus on defining your domain logic.

Features

• Unified Execution Model: Define work once, run on GPU or CPU
• Automatic Batch Planning: Byte-based batching optimized for VRAM constraints
• GPU Fallback: Graceful degradation to CPU when GPU execution fails
• Memory Management: Automatic VRAM probing and budget-aware scheduling
• Telemetry: Built-in performance tracking and reporting
• Configurable: JSON configuration or programmatic setup
• Feature Flags: Optional GPU support for CPU-only deployments

Quick Start

Add to your Cargo.toml:

[dependencies]

cgpu = "version"

CPU-Only Example

use cgpu::*;

#[derive(Clone)]
struct Item(u32);

impl WorkItem for Item {
    fn bytes_in(&self) -> usize { 4 }
    fn bytes_out(&self) -> usize { 4 }
}

struct DoubleJob {
    items: Vec<Item>,
}

impl GpuJob for DoubleJob {
    type Item = Item;
    type Output = u32;

    fn label(&self) -> &'static str { "double" }
    fn items(&self) -> &[Self::Item] { &self.items }

    fn encode_batch(&self, _span: &BatchSpan) -> Result<EncodedBatch, JobError> {
        // Return error to force CPU fallback
        Err(JobError::EncodingFailed("Using CPU path".into()))
    }

    fn decode_batch(&self, _bytes: &[u8], _span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        Ok(Vec::new())
    }

    fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        Ok(self.items[span.range()].iter().map(|item| item.0 * 2).collect())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut config = ExecutorConfig::default();
    config.execution_mode = ExecutionMode::PreferCpu;

    let mut executor = GpuExecutor::with_config(config)?;
    let job = DoubleJob {
        items: vec![Item(1), Item(2), Item(3)],
    };
    
    let report = executor.execute(&job)?;
    assert_eq!(report.outputs, vec![2, 4, 6]);
    
    println!("Execution path: {:?}", report.execution_path);
    println!("Total time: {:.2}ms", report.total_ms);
    
    Ok(())
}

GPU-Enabled Example

use cgpu::*;

struct GpuDoubleJob {
    items: Vec<u32>,
}

impl WorkItem for u32 {
    fn bytes_in(&self) -> usize { 4 }
    fn bytes_out(&self) -> usize { 4 }
}

impl GpuJob for GpuDoubleJob {
    type Item = u32;
    type Output = u32;

    fn label(&self) -> &'static str { "gpu-double" }
    fn items(&self) -> &[Self::Item] { &self.items }

    fn encode_batch(&self, span: &BatchSpan) -> Result<EncodedBatch, JobError> {
        let items = &self.items[span.range()];
        let input_bytes: Vec<u8> = items.iter()
            .flat_map(|x| x.to_ne_bytes())
            .collect();
        let output_size = items.len() * 4;

        let wgsl = r#"
@group(0) @binding(0)
var<storage, read> input: array<u32>;

@group(0) @binding(1)
var<storage, read_write> output: array<u32>;

@compute @workgroup_size(64, 1, 1)
fn main(@builtin(global_invocation_id) global_id: vec3<u32>) {
    if (global_id.x < arrayLength(&input)) {
        output[global_id.x] = input[global_id.x] * 2u;
    }
}
"#;

        let dispatch_x = ((items.len() as u32 + 63) / 64).max(1);

        Ok(EncodedBatch::new()
            .with_label("double-batch")
            .with_wgsl("double-shader", wgsl)
            .add_buffer(EncodedBuffer::storage_read_only(0, input_bytes))
            .add_buffer(
                EncodedBuffer::storage_read_write(1, BufferRole::Output, vec![0u8; output_size])
                    .readback(true)
            )
            .with_dispatch(dispatch_x, 1, 1))
    }

    fn decode_batch(&self, bytes: &[u8], span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        let count = span.len();
        let mut results = Vec::with_capacity(count);
        
        for i in 0..count {
            let offset = i * 4;
            if offset + 4 <= bytes.len() {
                let mut buf = [0u8; 4];
                buf.copy_from_slice(&bytes[offset..offset + 4]);
                results.push(u32::from_ne_bytes(buf));
            }
        }
        
        Ok(results)
    }

    fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<u32>, JobError> {
        Ok(self.items[span.range()].iter().map(|x| x * 2).collect())
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut executor = GpuExecutor::new()?;
    
    let job = GpuDoubleJob {
        items: (0..1000).collect(),
    };
    
    let report = executor.execute(&job)?;
    println!("Processed {} items via {:?}", 
             report.outputs.len(), 
             report.execution_path);
    
    Ok(())
}

Configuration

JSON Configuration

Create a cgpu.config.json file in your project root:

{
  "vram_override": null,
  "memory_fill_ratio": 0.9,
  "min_batch_bytes": 4096,
  "max_batch_bytes": 268435456,
  "execution_mode": "auto",
  "cpu_threads": null,
  "parallel_fallback": true,
  "shader_cache": true,
  "shader_optimization": "performance",
  "enable_telemetry": true,
  "telemetry_sink": "log"
}

Config Fields

The executor searches for this file in:

1. Current working directory
2. Crate directory
3. Workspace parent directory

Programmatic Configuration

let mut config = ExecutorConfig::default();
config.memory_fill_ratio = 0.85;
config.execution_mode = ExecutionMode::PreferGpu;
config.enable_telemetry = false;

let executor = GpuExecutor::with_config(config)?;

Environment Variables

• REV_GPU_AVAILABLE_BYTES: Override detected VRAM size

Architecture

Core Concepts

WorkItem: Describes a unit of work with byte size estimates

pub trait WorkItem: Send + Sync {
    fn bytes_in(&self) -> usize;
    fn bytes_out(&self) -> usize;
}

GpuJob: Defines how to encode/decode work for GPU execution

pub trait GpuJob: Send + Sync {
    type Item: WorkItem;
    type Output: Send;
    
    fn encode_batch(&self, span: &BatchSpan) -> Result<EncodedBatch, JobError>;
    fn decode_batch(&self, bytes: &[u8], span: &BatchSpan) -> Result<Vec<Self::Output>, JobError>;
    fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<Self::Output>, JobError>;
}

GpuExecutor: Manages execution, batching, and resource allocation

Execution Flow

1. Batch Planning: Items are grouped into batches based on byte size and VRAM budget
2. Encoding: Each batch is encoded into WGSL shaders and storage buffers
3. Execution: Batches run on GPU (with optional CPU fallback) or CPU
4. Decoding: GPU output bytes are converted back to domain types
5. Reporting: Structured JobReport with timing and execution path info

Execution Paths

• GpuOnly: All batches executed on GPU
• GpuWithFallback: Some batches fell back to CPU after GPU failure
• CpuOnly: All batches executed on CPU
• Mixed: Combination of GPU and CPU execution

Feature Flags

CPU-Only Build

cargo build --no-default-features

Full GPU Build

cargo build

Testing

# Format check

cargo fmt


# Type check

cargo check


# CPU-only tests

cargo test -p cgpu --no-default-features


# Full tests (requires GPU)

cargo test -p cgpu

Performance Considerations

• Batch Size: Larger batches reduce overhead but increase memory usage
• VRAM Budget: Configure memory_fill_ratio to leave headroom for other GPU resources
• Shader Caching: Keep shader_cache enabled for resource reuse and generated shader workflows
• Telemetry: Disable in production for minimal overhead

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Contributing

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.