# CLAUDE.md - Development Notes for bevy-sensor
## Project Overview
A Bevy library and CLI that captures multi-view images of 3D OBJ models (YCB dataset) for sensor simulation. This project produces comparable results to the [Thousand Brains Project (TBP)](https://github.com/thousandbrainsproject/tbp.monty) habitat sensor for use in the neocortx Rust-based implementation.
## Quick Reference
```bash
# Build
cargo build --release
# Run tests (69 tests)
cargo test
# Run (requires GPU or proper software rendering)
cargo run --release
# Headless with software rendering (limited - PBR shaders may fail)
LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=llvmpipe \
xvfb-run -a -s "-screen 0 1280x1024x24" \
cargo run --release
```
## Architecture
### Library API (`src/lib.rs`)
The library exports these types for use by neocortx:
```rust
use bevy_sensor::{
// Headless Rendering API (NEW)
RenderConfig, // Render settings (resolution, lighting, depth)
RenderOutput, // RGBA + depth buffer output
LightingConfig, // Configurable lighting
CameraIntrinsics, // Camera parameters for 3D projection
render_to_buffer, // Render single viewpoint to memory
render_all_viewpoints, // Batch render all viewpoints
// File-based Capture (Legacy)
SensorConfig, // Full capture configuration
ViewpointConfig, // Camera viewpoint settings
ObjectRotation, // Object rotation (Euler angles)
generate_viewpoints, // Generate camera transforms
CaptureCamera, // Marker component for camera
CaptureTarget, // Marker component for target object
};
// YCB utilities
use bevy_sensor::ycb::{
download_models, // Async download of YCB models
models_exist, // Check if models downloaded
object_mesh_path, // Get OBJ file path
object_texture_path, // Get texture path
Subset, // Representative, Ten, All
REPRESENTATIVE_OBJECTS,
TEN_OBJECTS,
};
```
### Headless Rendering API (NEW)
Render directly to memory for neocortx integration:
```rust
use bevy_sensor::{render_to_buffer, RenderConfig, ViewpointConfig, ObjectRotation};
use std::path::Path;
// TBP-compatible 64x64 sensor
let config = RenderConfig::tbp_default();
let viewpoints = bevy_sensor::generate_viewpoints(&ViewpointConfig::default());
// Render single viewpoint
let output = render_to_buffer(
Path::new("/tmp/ycb/003_cracker_box"),
&viewpoints[0],
&ObjectRotation::identity(),
&config,
)?;
// Access rendered data
let rgba: &[u8] = &output.rgba; // 64*64*4 bytes
let depth: &[f64] = &output.depth; // 64*64 f64 (meters, TBP precision)
let rgb_image = output.to_rgb_image(); // Vec<Vec<[u8; 3]>> for neocortx
let depth_image = output.to_depth_image(); // Vec<Vec<f64>> for neocortx
```
**RenderConfig options:**
| `RenderConfig::tbp_default()` | 64×64 | TBP-compatible sensor |
| `RenderConfig::preview()` | 256×256 | Debugging/visualization |
| `RenderConfig::high_res()` | 512×512 | Detailed captures |
**LightingConfig options:**
| `LightingConfig::default()` | Standard 3-point lighting |
| `LightingConfig::bright()` | High visibility |
| `LightingConfig::soft()` | Soft, even lighting |
| `LightingConfig::unlit()` | Ambient only, no shadows |
**CameraIntrinsics:**
```rust
let intrinsics = config.intrinsics();
// intrinsics.focal_length: [f64; 2] - (fx, fy) in pixels (TBP precision)
// intrinsics.principal_point: [f64; 2] - (cx, cy) center (TBP precision)
// intrinsics.image_size: [u32; 2] - (width, height)
// Project 3D to 2D (accepts Bevy Vec3, returns f64)
let pixel = intrinsics.project(point_3d); // Option<[f64; 2]>
// Unproject 2D + depth to 3D (all f64 for TBP precision)
let point = intrinsics.unproject([32.0, 32.0], depth); // [f64; 3]
```
> **Note:** Full GPU rendering requires a display (X11/Wayland). The depth buffer
> now returns real per-pixel depth values from GPU readback (Bevy 0.15+).
### Object Rotation (`ObjectRotation`)
Matches TBP benchmark Euler angle format `[pitch, yaw, roll]`:
```rust
// TBP benchmark: 3 rotations (used for quick experiments)
ObjectRotation::tbp_benchmark_rotations()
// → [[0,0,0], [0,90,0], [0,180,0]]
// TBP known orientations: 14 rotations (6 faces + 8 corners)
ObjectRotation::tbp_known_orientations()
// → 14 orientations used during TBP training
```
### Batch Rendering API (NEW)
For rendering 10+ viewpoints efficiently, use the batch API to eliminate subprocess and app initialization overhead. Achieves 10-50x speedup compared to sequential `render_to_buffer()` calls.
**Quick Example:**
```rust
use bevy_sensor::{
create_batch_renderer, queue_render_request, render_next_in_batch,
BatchRenderRequest, BatchRenderConfig, RenderConfig, ObjectRotation,
};
use std::path::PathBuf;
// Create renderer once
let mut renderer = create_batch_renderer(&BatchRenderConfig::default())?;
// Queue renders
for viewpoint in viewpoints {
queue_render_request(&mut renderer, BatchRenderRequest {
object_dir: PathBuf::from("/tmp/ycb/003_cracker_box"),
viewpoint,
object_rotation: ObjectRotation::identity(),
render_config: RenderConfig::tbp_default(),
})?;
}
// Execute and collect results
let mut results = Vec::new();
loop {
match render_next_in_batch(&mut renderer, 500)? {
Some(output) => results.push(output),
None => break,
}
}
```
**Convenience Function (for simple cases):**
```rust
use bevy_sensor::{render_batch, batch::BatchRenderRequest, BatchRenderConfig};
let results = render_batch(requests, &BatchRenderConfig::default())?;
```
**Performance Comparison:**
| `render_to_buffer()` × 72 (sequential) | ~30-50s |
| `render_all_viewpoints()` | ~25-40s |
| `render_batch()` (current) | ~20-35s |
| `render_batch()` (future: persistent app) | ~1-3s |
**When to Use:**
- ✅ Batch API: 10+ renders (any object/config mix)
- ✅ Batch API: Rendering entire dataset iterations
- ✅ Single `render_to_buffer()`: 1-3 renders or quick tests
- ✅ `render_all_viewpoints()`: Legacy code (still works)
**API Types:**
```rust
pub struct BatchRenderRequest {
pub object_dir: PathBuf, // Path to YCB object
pub viewpoint: Transform, // Camera position
pub object_rotation: ObjectRotation,
pub render_config: RenderConfig,
}
pub struct BatchRenderOutput {
pub request: BatchRenderRequest,
pub rgba: Vec<u8>, // RGBA pixels
pub depth: Vec<f64>, // Depth in meters (f64)
pub status: RenderStatus, // Success, PartialFailure, Failed
// ... other fields
}
pub enum RenderStatus {
Success, // RGBA + depth complete
PartialFailure, // RGBA ok, depth missing
Failed, // Render failed
}
```
**neocortx Integration:**
```rust
// Render 72 observations for benchmark
let requests: Vec<_> = rotations.iter()
.flat_map(|rot| viewpoints.iter().map(move |vp| {
BatchRenderRequest {
object_dir: object_dir.clone(),
viewpoint: *vp,
object_rotation: rot.clone(),
render_config: RenderConfig::tbp_default(),
}
}))
.collect();
let outputs = render_batch(requests, &BatchRenderConfig::default())?;
// Convert to neocortx observation format
for output in outputs {
let rgb_image: Vec<Vec<[u8; 3]>> = output.to_rgb_image();
let depth_image: Vec<Vec<f64>> = output.to_depth_image();
// Send to neocortx sensorimotor system...
}
```
See `examples/batch_render_neocortx.rs` for a complete working example.
### Viewpoint Generation
Uses **spherical coordinates** matching TBP habitat sensor behavior:
```rust
struct ViewpointConfig {
radius: f32, // Distance from object (default: 0.5m)
yaw_count: usize, // Horizontal positions (default: 8)
pitch_angles_deg: Vec<f32>, // Elevations (default: [-30°, 0°, +30°])
}
```
**Spherical to Cartesian conversion (Y-up):**
```
x = radius * cos(pitch) * sin(yaw)
y = radius * sin(pitch)
z = radius * cos(pitch) * cos(yaw)
```
### Capture Configurations
| `SensorConfig::default()` | 1 | 24 | 24 |
| `SensorConfig::tbp_benchmark()` | 3 | 24 | 72 |
| `SensorConfig::tbp_full_training()` | 14 | 24 | 336 |
### Capture State Machine (`src/main.rs`)
```
SetupRotation → SetupView → WaitSettle (10 frames) → Capture → WaitSave (200 frames) → loop
```
Output: `capture_{rot}_{view}.png` (e.g., `capture_0_0.png` through `capture_2_23.png`)
## TBP Habitat Sensor Alignment
| Quaternion rotation | `ObjectRotation::to_quat()` |
| `look_up` / `look_down` | Pitch angles: -30°, 0°, +30° |
| `turn_left` / `turn_right` | 8 yaw positions @ 45° intervals |
| Object rotations [0,0,0], [0,90,0], [0,180,0] | `ObjectRotation::tbp_benchmark_rotations()` |
| 14 known orientations | `ObjectRotation::tbp_known_orientations()` |
| Spherical radius | Configurable (default 0.5m) |
### TBP Reference Implementation
- **Sensors**: `tbp/monty/simulators/habitat/sensors.py`
- Uses quaternion (w,x,y,z) format, default identity: (1, 0, 0, 0)
- Position relative to HabitatAgent
- **Actions**: `tbp/monty/frameworks/actions/actions.py`
- `LookDown`, `LookUp`: rotation_degrees parameter
- `TurnLeft`, `TurnRight`: yaw rotation
- `SetYaw`, `SetAgentPitch`: absolute rotations
- **Benchmarks**:
- 3 known rotations for quick tests: [0,0,0], [0,90,0], [0,180,0]
- 14 known orientations for full training (cube faces + corners)
- 10 random rotations for generalization testing
## YCB Dataset Setup
**Programmatic download (via ycbust library):**
```rust
use bevy_sensor::ycb::{download_models, Subset, models_exist};
// Check if models already exist
if !models_exist("/tmp/ycb") {
// Download representative subset (3 objects) - async
download_models("/tmp/ycb", Subset::Representative).await?;
}
// Get paths to specific object files
let mesh = bevy_sensor::ycb::object_mesh_path("/tmp/ycb", "003_cracker_box");
let texture = bevy_sensor::ycb::object_texture_path("/tmp/ycb", "003_cracker_box");
```
**Available subsets:**
- `Subset::Representative` - 3 objects (quick testing)
- `Subset::Ten` - 10 objects (TBP benchmark subset)
- `Subset::All` - All 77 YCB objects
The `assets/ycb` symlink points to `/tmp/ycb`.
## Dependencies
```toml
bevy = { version = "0.15", default-features = false, features = [
"bevy_asset",
"bevy_core_pipeline",
"bevy_pbr",
"bevy_render",
"bevy_scene",
"bevy_state",
"bevy_winit",
"png",
"x11",
"tonemapping_luts",
"ktx2",
"zstd",
] }
bevy_obj = { version = "0.15", features = ["scene"] }
ycbust = "0.2.3"
```
### Bevy 0.15 Upgrade (December 2024)
The upgrade from Bevy 0.11 to 0.15 enabled real GPU depth buffer readback:
**Key Changes:**
- `ViewDepthTexture` replaces `ViewPrepassTextures` for depth access
- `Screenshot` entity + observer pattern replaces `ScreenshotManager`
- Component-based spawning (`Camera3d`, `PointLight`) replaces bundles
- `Mesh3d` and `MeshMaterial3d<M>` wrappers for mesh/material handles
- `SceneRoot(handle)` replaces `SceneBundle`
- MSAA must be disabled (`Msaa::Off`) for depth texture copy
**Depth Buffer Implementation:**
- Uses `ViewDepthTexture` from `bevy::render::view`
- Reverse-Z depth converted to linear meters via `reverse_z_to_linear_depth()`
- Custom render graph node (`DepthReadbackNode`) copies depth after main pass
- 256-byte row alignment for GPU buffer mapping
## WebGPU Backend Support
bevy-sensor now includes automatic cross-platform rendering backend selection via the `backend` module:
### Automatic Platform Detection
- **Linux**: Vulkan (primary), OpenGL (fallback), WebGPU (as-needed)
- **WSL2**: WebGPU (primary - recommended for WSL2), OpenGL (fallback)
- **macOS**: Metal (primary), OpenGL (fallback)
- **Windows**: Direct3D 12 (primary), Vulkan (fallback), WebGPU (as-needed)
### Using WebGPU Backend
```rust
use bevy_sensor::backend::{BackendConfig, RenderBackend};
// Use WebGPU explicitly for maximum compatibility
let config = BackendConfig {
preferred: Some(RenderBackend::WebGPU),
fallbacks: vec![RenderBackend::OpenGL],
debug_logging: false,
force_headless: true,
};
config.apply_env();
// Now run your render functions - they'll use WebGPU
```
### Environment Variable Override
```bash
# Force WebGPU backend
export WGPU_BACKEND=webgpu
cargo run --release
# Other options: vulkan, metal, dx12, gl, webgpu, webgl2, auto
```
### WSL2 Optimization
WSL2 environments now default to WebGPU backend (previously required pre-rendered fixtures):
```bash
# This now works on WSL2 (auto-selects WebGPU)
cargo run --example batch_render_neocortx --release
```
**Note**: WebGPU backend may have slightly lower performance than native backends but offers excellent compatibility across platforms.
## Known Limitations
1. **WSL2 GPU rendering** (PARTIALLY FIXED):
- WSL2 NOW has GPU access (nvidia-smi shows RTX 4090, CUDA 12.8 available)
- **Issue**: Bevy 0.15 GPU detection fails on WSL2 despite drivers being installed
- **Workaround**: Use pre-rendered fixtures for CI/CD (see below)
- **Note**: This is a Bevy + WSL2 compatibility issue, may be fixed in Bevy 0.16+
- **Solution for native Linux**: Integration tests work perfectly with GPU
2. **Software rendering (llvmpipe)**:
- Must disable tonemapping (`Tonemapping::None`) on the camera
- PBR shaders may fail with `gsamplerCubeArrayShadow` error
- Xvfb may crash due to NVIDIA driver conflicts in WSL2
3. **Texture loading with bevy_obj**:
- The bevy_obj scene loader has "limited MTL support"
- **Workaround**: Load textures manually with `asset_server.load()`
- MTL files with trailing spaces may cause loading failures
4. **Asset path**: When running binary from `target/release/`, assets must be in `target/release/assets/`
## Pre-rendered Fixtures for CI/CD
For environments without GPU rendering (WSL2, CI servers, Docker), use pre-rendered fixtures:
### Generating Fixtures
Run on a machine with a working display (Linux desktop with GPU):
```bash
# Generate test fixtures for CI/CD
cargo run --bin prerender -- --output-dir test_fixtures/renders
# Custom objects
cargo run --bin prerender -- --objects "003_cracker_box,005_tomato_soup_can,006_mustard_bottle"
```
This creates:
- `test_fixtures/renders/metadata.json` - Dataset configuration
- `test_fixtures/renders/{object_id}/` - Per-object renders
- `r{N}_v{M}.png` - RGBA images
- `r{N}_v{M}.depth` - Depth data (binary f32)
- `index.json` - Per-object metadata
### Loading Fixtures in Tests
```rust
use bevy_sensor::fixtures::TestFixtures;
// Load pre-rendered data
let fixtures = TestFixtures::load("test_fixtures/renders")?;
// Get a specific render
let output = fixtures.get_render("003_cracker_box", 0, 0)?; // rotation 0, viewpoint 0
// Use like normal RenderOutput
let rgb = output.to_rgb_image();
let depth = output.to_depth_image();
```
### neocortx Integration with Fixtures
```rust
// In neocortx tests, prefer fixtures for CI/CD compatibility
#[cfg(test)]
mod tests {
use bevy_sensor::fixtures::TestFixtures;
#[test]
fn test_sensor_with_fixtures() {
let fixtures = TestFixtures::load("test_fixtures/renders")
.expect("Pre-rendered fixtures required. Run: cargo run --bin prerender");
// Test with pre-rendered data
for (object_id, rotation_idx, viewpoint_idx, output) in fixtures.iter_renders() {
// Process render output...
}
}
}
```
### WSL2 Workarounds
If you need live rendering on WSL2, these options may work (unreliable):
1. **VcXsrv on Windows**: Install VcXsrv, set `DISPLAY=172.x.x.x:0`
2. **X410 on Windows**: Commercial X server with better GPU support
3. **Native Linux VM**: Use VirtualBox/VMware with GPU passthrough
**Recommended**: Generate fixtures on a Linux machine and commit to repo for CI/CD use.
## Viewpoint Coordinate Reference
```
View 0-7: pitch=-30° (below), yaw=0°-315° @ 45° steps, Y=-0.250
View 8-15: pitch=0° (level), yaw=0°-315° @ 45° steps, Y=0.000
View 16-23: pitch=+30° (above), yaw=0°-315° @ 45° steps, Y=+0.250
```
## Usage from neocortx
```rust
use bevy_sensor::{SensorConfig, ObjectRotation, ViewpointConfig};
use bevy_sensor::ycb::{download_models, Subset, models_exist};
// Ensure YCB models are available
if !models_exist("/tmp/ycb") {
download_models("/tmp/ycb", Subset::Representative).await?;
}
// Create capture config
let config = SensorConfig {
viewpoints: ViewpointConfig {
radius: 0.5,
yaw_count: 8,
pitch_angles_deg: vec![-30.0, 0.0, 30.0],
},
object_rotations: ObjectRotation::tbp_benchmark_rotations(),
output_dir: "./captures".to_string(),
filename_pattern: "ycb_{rot}_{view}.png".to_string(),
};
println!("Total captures: {}", config.total_captures()); // 72
```
## Related Projects
- **neocortx**: Rust-based Thousand Brains implementation (main project)
- **tbp.monty**: Original Python implementation by Thousand Brains Project
- **tbp.tbs_sensorimotor_intelligence**: TBP experiment configs
- **ycbust**: YCB dataset downloader (used as library dependency)
## API Parity with neocortx
bevy-sensor provides complete API parity with neocortx's `bevy_simulator` module:
| RGBA image data | `RenderOutput.rgba: Vec<u8>` | ✅ |
| Depth buffer (meters, f64) | `RenderOutput.depth: Vec<f64>` | ✅ |
| Camera intrinsics (f64) | `CameraIntrinsics` struct | ✅ |
| Camera position | `RenderOutput.camera_transform` | ✅ |
| Object rotation (f64) | `RenderOutput.object_rotation` | ✅ |
| RGB image format | `to_rgb_image() → Vec<Vec<[u8; 3]>>` | ✅ |
| Depth image format (f64) | `to_depth_image() → Vec<Vec<f64>>` | ✅ |
| TBP viewpoints (24) | `ViewpointConfig::default()` | ✅ |
| TBP benchmark rotations (3) | `ObjectRotation::tbp_benchmark_rotations()` | ✅ |
| TBP full rotations (14) | `ObjectRotation::tbp_known_orientations()` | ✅ |
| YCB model download | `ycb::download_models()` | ✅ |
| Resolution config | `RenderConfig` (64×64, 256×256, 512×512) | ✅ |
**neocortx integration example:**
```rust
use bevy_sensor::{render_to_buffer, RenderConfig, ViewpointConfig, ObjectRotation};
// Render and convert to neocortx formats
let output = render_to_buffer(object_dir, &viewpoint, &rotation, &config)?;
let rgb_image: Vec<Vec<[u8; 3]>> = output.to_rgb_image(); // VisionObservation.image
let depth_image: Vec<Vec<f64>> = output.to_depth_image(); // For surface normal (f64 precision)
let intrinsics = output.intrinsics; // CameraIntrinsics (f64)
```
## Release Flow & Versioning
**Version bumps are handled automatically by release-please.** Do NOT manually bump version numbers in `Cargo.toml`.
### Conventional Commits
This project uses [Conventional Commits](https://www.conventionalcommits.org/) for automated changelog and version management:
- `feat:` - New features (minor version bump)
- `fix:` - Bug fixes (patch version bump)
- `feat!:` or `BREAKING CHANGE:` - Breaking changes (major version bump)
- `docs:`, `chore:`, `refactor:`, `test:` - No version bump
### Release Process
1. Merge PRs with conventional commit messages to `main`
2. release-please creates/updates a "Release PR" with:
- Updated `Cargo.toml` version
- Updated `CHANGELOG.md`
3. Merge the Release PR to trigger:
- Git tag creation
- GitHub release
- crates.io publish (if configured)
**Note:** Git tags are created asynchronously by GitHub Actions (30-60 seconds). Don't repeatedly `git fetch` after merging the release PR—the tag will be available once the workflow completes. Check status with `gh run list --workflow CI`.
### Breaking Changes
When making breaking API changes (like f32 → f64 migration):
1. Use `feat!:` prefix in commit message
2. Include `BREAKING CHANGE:` section in commit body
3. Document migration in PR description
## Resources
- [TBP Benchmark Experiments](https://thousandbrainsproject.readme.io/docs/benchmark-experiments)
- [TBP GitHub](https://github.com/thousandbrainsproject/tbp.monty)
- [YCB Object Dataset](https://www.ycbbenchmarks.com/)