# cgpu
[](https://crates.io/crates/cgpu)
**Reusable CPU/GPU compute execution layer for Rust.**
`cgpu` provides a unified interface for running compute jobs that can execute on the GPU via `wgpu` or fall back to CPU execution. It handles batch planning, memory management, shader compilation, and telemetry automatically, letting you focus on defining your domain logic.
## Features
- **Unified Execution Model**: Define work once, run on GPU or CPU
- **Automatic Batch Planning**: Byte-based batching optimized for VRAM constraints
- **GPU Fallback**: Graceful degradation to CPU when GPU execution fails
- **Memory Management**: Automatic VRAM probing and budget-aware scheduling
- **Telemetry**: Built-in performance tracking and reporting
- **Configurable**: JSON configuration or programmatic setup
- **Feature Flags**: Optional GPU support for CPU-only deployments
## Quick Start
Add to your `Cargo.toml`:
```toml
[dependencies]
cgpu = "version"
```
### CPU-Only Example
```rust
use cgpu::*;
#[derive(Clone)]
struct Item(u32);
impl WorkItem for Item {
fn bytes_in(&self) -> usize { 4 }
fn bytes_out(&self) -> usize { 4 }
}
struct DoubleJob {
items: Vec<Item>,
}
impl GpuJob for DoubleJob {
type Item = Item;
type Output = u32;
fn label(&self) -> &'static str { "double" }
fn items(&self) -> &[Self::Item] { &self.items }
fn encode_batch(&self, _span: &BatchSpan) -> Result<EncodedBatch, JobError> {
// Return error to force CPU fallback
Err(JobError::EncodingFailed("Using CPU path".into()))
}
fn decode_batch(&self, _bytes: &[u8], _span: &BatchSpan) -> Result<Vec<u32>, JobError> {
Ok(Vec::new())
}
fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<u32>, JobError> {
Ok(self.items[span.range()].iter().map(|item| item.0 * 2).collect())
}
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut config = ExecutorConfig::default();
config.execution_mode = ExecutionMode::PreferCpu;
let mut executor = GpuExecutor::with_config(config)?;
let job = DoubleJob {
items: vec![Item(1), Item(2), Item(3)],
};
let report = executor.execute(&job)?;
assert_eq!(report.outputs, vec![2, 4, 6]);
println!("Execution path: {:?}", report.execution_path);
println!("Total time: {:.2}ms", report.total_ms);
Ok(())
}
```
### GPU-Enabled Example
```rust
use cgpu::*;
struct GpuDoubleJob {
items: Vec<u32>,
}
impl WorkItem for u32 {
fn bytes_in(&self) -> usize { 4 }
fn bytes_out(&self) -> usize { 4 }
}
impl GpuJob for GpuDoubleJob {
type Item = u32;
type Output = u32;
fn label(&self) -> &'static str { "gpu-double" }
fn items(&self) -> &[Self::Item] { &self.items }
fn encode_batch(&self, span: &BatchSpan) -> Result<EncodedBatch, JobError> {
let items = &self.items[span.range()];
let input_bytes: Vec<u8> = items.iter()
.flat_map(|x| x.to_ne_bytes())
.collect();
let output_size = items.len() * 4;
let wgsl = r#"
@group(0) @binding(0)
var<storage, read> input: array<u32>;
@group(0) @binding(1)
var<storage, read_write> output: array<u32>;
@compute @workgroup_size(64, 1, 1)
fn main(@builtin(global_invocation_id) global_id: vec3<u32>) {
if (global_id.x < arrayLength(&input)) {
output[global_id.x] = input[global_id.x] * 2u;
}
}
"#;
let dispatch_x = ((items.len() as u32 + 63) / 64).max(1);
Ok(EncodedBatch::new()
.with_label("double-batch")
.with_wgsl("double-shader", wgsl)
.add_buffer(EncodedBuffer::storage_read_only(0, input_bytes))
.add_buffer(
EncodedBuffer::storage_read_write(1, BufferRole::Output, vec![0u8; output_size])
.readback(true)
)
.with_dispatch(dispatch_x, 1, 1))
}
fn decode_batch(&self, bytes: &[u8], span: &BatchSpan) -> Result<Vec<u32>, JobError> {
let count = span.len();
let mut results = Vec::with_capacity(count);
for i in 0..count {
let offset = i * 4;
if offset + 4 <= bytes.len() {
let mut buf = [0u8; 4];
buf.copy_from_slice(&bytes[offset..offset + 4]);
results.push(u32::from_ne_bytes(buf));
}
}
Ok(results)
}
fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<u32>, JobError> {
Ok(self.items[span.range()].iter().map(|x| x * 2).collect())
}
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut executor = GpuExecutor::new()?;
let job = GpuDoubleJob {
items: (0..1000).collect(),
};
let report = executor.execute(&job)?;
println!("Processed {} items via {:?}",
report.outputs.len(),
report.execution_path);
Ok(())
}
```
## Configuration
### JSON Configuration
Create a `cgpu.config.json` file in your project root:
```json
{
"vram_override": null,
"memory_fill_ratio": 0.9,
"min_batch_bytes": 4096,
"max_batch_bytes": 268435456,
"execution_mode": "auto",
"cpu_threads": null,
"parallel_fallback": true,
"shader_cache": true,
"shader_optimization": "performance",
"enable_telemetry": true,
"telemetry_sink": "log"
}
```
### Config Fields
| `vram_override` | `null` | Optional memory budget override in bytes. Use this when you want deterministic planning or when automatic VRAM probing is not trustworthy on the target machine. `null` leaves the executor on its normal budget path. |
| `memory_fill_ratio` | `0.9` | Fraction of the available memory budget that batching is allowed to use. Keep this below `1.0` so command buffers, staging buffers, desktop compositors, and other GPU users still have headroom. The practical GPU budget path expects the safe range around `0.90` to `0.95`. |
| `min_batch_bytes` | `4096` | Minimum estimated batch size before `ExecutionMode::Auto` considers GPU execution worthwhile. Smaller batches usually stay on CPU because upload, dispatch, and readback overhead can dominate the work. |
| `max_batch_bytes` | `268435456` | Hard upper bound for one planned batch. This protects the executor from creating oversized buffers even if the reported or overridden memory budget is large. |
| `execution_mode` | `"auto"` | Selects the execution policy. `"auto"` chooses GPU only when available and the batch is large enough, `"prefer_gpu"` tries GPU first and falls back to CPU, `"prefer_cpu"` always uses CPU fallback, and `"gpu_only"` returns an error instead of falling back. |
| `cpu_threads` | `null` | Optional CPU worker count for CPU execution and fallback policy. `null` means use the platform/default parallelism. The current generic executor keeps this value in config so fallback scheduling can be tuned without changing the public API. |
| `parallel_fallback` | `true` | Allows CPU fallback work to be parallelized by the fallback policy. Keep it enabled for throughput-oriented jobs; disable it when a caller needs strictly serial fallback behavior. |
| `shader_cache` | `true` | Enables the shader/resource cache policy. The resource cache is active today; shader and pipeline reuse are represented by this knob so generated shader workflows can opt into caching as that layer grows. |
| `shader_optimization` | `"performance"` | Declares the shader generation preference. `"debug"` favors fast iteration and readable generated WGSL, `"performance"` is the default throughput profile, and `"size"` is for smaller generated shader bodies or binary pressure. |
| `enable_telemetry` | `true` | Turns phase timing and byte accounting on or off. Disable it when the caller wants the smallest possible bookkeeping overhead. |
| `telemetry_sink` | `"log"` | Chooses where telemetry is sent. JSON config currently uses `"log"`; programmatic config can also use callback or channel buffer sinks. |
The executor searches for this file in:
1. Current working directory
2. Crate directory
3. Workspace parent directory
### Programmatic Configuration
```rust
let mut config = ExecutorConfig::default();
config.memory_fill_ratio = 0.85;
config.execution_mode = ExecutionMode::PreferGpu;
config.enable_telemetry = false;
let executor = GpuExecutor::with_config(config)?;
```
### Environment Variables
- `REV_GPU_AVAILABLE_BYTES`: Override detected VRAM size
## Architecture
### Core Concepts
**WorkItem**: Describes a unit of work with byte size estimates
```rust
pub trait WorkItem: Send + Sync {
fn bytes_in(&self) -> usize;
fn bytes_out(&self) -> usize;
}
```
**GpuJob**: Defines how to encode/decode work for GPU execution
```rust
pub trait GpuJob: Send + Sync {
type Item: WorkItem;
type Output: Send;
fn encode_batch(&self, span: &BatchSpan) -> Result<EncodedBatch, JobError>;
fn decode_batch(&self, bytes: &[u8], span: &BatchSpan) -> Result<Vec<Self::Output>, JobError>;
fn cpu_fallback(&self, span: &BatchSpan) -> Result<Vec<Self::Output>, JobError>;
}
```
**GpuExecutor**: Manages execution, batching, and resource allocation
### Execution Flow
1. **Batch Planning**: Items are grouped into batches based on byte size and VRAM budget
2. **Encoding**: Each batch is encoded into WGSL shaders and storage buffers
3. **Execution**: Batches run on GPU (with optional CPU fallback) or CPU
4. **Decoding**: GPU output bytes are converted back to domain types
5. **Reporting**: Structured `JobReport` with timing and execution path info
### Execution Paths
- `GpuOnly`: All batches executed on GPU
- `GpuWithFallback`: Some batches fell back to CPU after GPU failure
- `CpuOnly`: All batches executed on CPU
- `Mixed`: Combination of GPU and CPU execution
## Feature Flags
| `gpu` | Enable GPU support via wgpu | ✓ |
| `telemetry` | Enable performance telemetry | ✓ |
| `shader` | Enable shader helpers | ✓ |
| `shader-gen` | Enable shader generation utilities | (via `shader`) |
| `async` | Enable async support with tokio | |
| `vulkan` | Vulkan backend support | |
| `dx12` | DirectX 12 backend support | |
| `metal` | Metal backend support (macOS/iOS) | |
| `webgl` | WebGL backend support | |
### CPU-Only Build
```bash
cargo build --no-default-features
```
### Full GPU Build
```bash
cargo build
```
## Testing
```bash
# Format check
cargo fmt
# Type check
cargo check
# CPU-only tests
cargo test -p cgpu --no-default-features
# Full tests (requires GPU)
cargo test -p cgpu
```
## Performance Considerations
- **Batch Size**: Larger batches reduce overhead but increase memory usage
- **VRAM Budget**: Configure `memory_fill_ratio` to leave headroom for other GPU resources
- **Shader Caching**: Keep `shader_cache` enabled for resource reuse and generated shader workflows
- **Telemetry**: Disable in production for minimal overhead
## License
Licensed under either of:
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT license ([LICENSE-MIT](LICENSE-MIT))
at your option.
## Contributing
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.