bevy-sensor 0.2.2

# CLAUDE.md - Development Notes for bevy-sensor

## Project Overview

A Bevy library and CLI that captures multi-view images of 3D OBJ models (YCB dataset) for sensor simulation. This project produces comparable results to the [Thousand Brains Project (TBP)](https://github.com/thousandbrainsproject/tbp.monty) habitat sensor for use in the neocortx Rust-based implementation.

## Quick Reference

```bash
# Build
cargo build --release

# Run tests (69 tests)
cargo test

# Run (requires GPU or proper software rendering)
cargo run --release

# Headless with software rendering (limited - PBR shaders may fail)
LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=llvmpipe \
  xvfb-run -a -s "-screen 0 1280x1024x24" \
  cargo run --release
```

## Architecture

### Library API (`src/lib.rs`)

The library exports these types for use by neocortx:

```rust
use bevy_sensor::{
    // Headless Rendering API (NEW)
    RenderConfig,        // Render settings (resolution, lighting, depth)
    RenderOutput,        // RGBA + depth buffer output
    LightingConfig,      // Configurable lighting
    CameraIntrinsics,    // Camera parameters for 3D projection
    render_to_buffer,    // Render single viewpoint to memory
    render_all_viewpoints, // Batch render all viewpoints

    // File-based Capture (Legacy)
    SensorConfig,        // Full capture configuration
    ViewpointConfig,     // Camera viewpoint settings
    ObjectRotation,      // Object rotation (Euler angles)
    generate_viewpoints, // Generate camera transforms
    CaptureCamera,       // Marker component for camera
    CaptureTarget,       // Marker component for target object
};

// YCB utilities
use bevy_sensor::ycb::{
    download_models,     // Async download of YCB models
    models_exist,        // Check if models downloaded
    object_mesh_path,    // Get OBJ file path
    object_texture_path, // Get texture path
    Subset,              // Representative, Ten, All
    REPRESENTATIVE_OBJECTS,
    TEN_OBJECTS,
};
```

### Headless Rendering API (NEW)

Render directly to memory for neocortx integration:

```rust
use bevy_sensor::{render_to_buffer, RenderConfig, ViewpointConfig, ObjectRotation};
use std::path::Path;

// TBP-compatible 64x64 sensor
let config = RenderConfig::tbp_default();
let viewpoints = bevy_sensor::generate_viewpoints(&ViewpointConfig::default());

// Render single viewpoint
let output = render_to_buffer(
    Path::new("/tmp/ycb/003_cracker_box"),
    &viewpoints[0],
    &ObjectRotation::identity(),
    &config,
)?;

// Access rendered data
let rgba: &[u8] = &output.rgba;           // 64*64*4 bytes
let depth: &[f32] = &output.depth;        // 64*64 floats (meters)
let rgb_image = output.to_rgb_image();    // Vec<Vec<[u8; 3]>> for neocortx
let depth_image = output.to_depth_image(); // Vec<Vec<f32>> for neocortx
```

**RenderConfig options:**

| Constructor | Resolution | Use case |
|-------------|-----------|----------|
| `RenderConfig::tbp_default()` | 64×64 | TBP-compatible sensor |
| `RenderConfig::preview()` | 256×256 | Debugging/visualization |
| `RenderConfig::high_res()` | 512×512 | Detailed captures |

**LightingConfig options:**

| Constructor | Description |
|-------------|-------------|
| `LightingConfig::default()` | Standard 3-point lighting |
| `LightingConfig::bright()` | High visibility |
| `LightingConfig::soft()` | Soft, even lighting |
| `LightingConfig::unlit()` | Ambient only, no shadows |

**CameraIntrinsics:**

```rust
let intrinsics = config.intrinsics();
// intrinsics.focal_length: [f32; 2]     - (fx, fy) in pixels
// intrinsics.principal_point: [f32; 2]  - (cx, cy) center
// intrinsics.image_size: [u32; 2]       - (width, height)

// Project 3D to 2D
let pixel = intrinsics.project(point_3d);  // Option<[f32; 2]>

// Unproject 2D + depth to 3D
let point = intrinsics.unproject([32.0, 32.0], depth);  // Vec3
```

> **Note:** Full GPU rendering requires a display (X11/Wayland). The depth buffer
> now returns real per-pixel depth values from GPU readback (Bevy 0.15+).

### Object Rotation (`ObjectRotation`)

Matches TBP benchmark Euler angle format `[pitch, yaw, roll]`:

```rust
// TBP benchmark: 3 rotations (used for quick experiments)
ObjectRotation::tbp_benchmark_rotations()
// → [[0,0,0], [0,90,0], [0,180,0]]

// TBP known orientations: 14 rotations (6 faces + 8 corners)
ObjectRotation::tbp_known_orientations()
// → 14 orientations used during TBP training
```

### Viewpoint Generation

Uses **spherical coordinates** matching TBP habitat sensor behavior:

```rust
struct ViewpointConfig {
    radius: f32,              // Distance from object (default: 0.5m)
    yaw_count: usize,         // Horizontal positions (default: 8)
    pitch_angles_deg: Vec<f32>, // Elevations (default: [-30°, 0°, +30°])
}
```

**Spherical to Cartesian conversion (Y-up):**
```
x = radius * cos(pitch) * sin(yaw)
y = radius * sin(pitch)
z = radius * cos(pitch) * cos(yaw)
```

### Capture Configurations

| Config | Rotations | Viewpoints | Total |
|--------|-----------|------------|-------|
| `SensorConfig::default()` | 1 | 24 | 24 |
| `SensorConfig::tbp_benchmark()` | 3 | 24 | 72 |
| `SensorConfig::tbp_full_training()` | 14 | 24 | 336 |

### Capture State Machine (`src/main.rs`)

```
SetupRotation → SetupView → WaitSettle (10 frames) → Capture → WaitSave (200 frames) → loop
```

Output: `capture_{rot}_{view}.png` (e.g., `capture_0_0.png` through `capture_2_23.png`)

## TBP Habitat Sensor Alignment

| TBP Feature | Bevy Implementation |
|-------------|---------------------|
| Quaternion rotation | `ObjectRotation::to_quat()` |
| `look_up` / `look_down` | Pitch angles: -30°, 0°, +30° |
| `turn_left` / `turn_right` | 8 yaw positions @ 45° intervals |
| Object rotations [0,0,0], [0,90,0], [0,180,0] | `ObjectRotation::tbp_benchmark_rotations()` |
| 14 known orientations | `ObjectRotation::tbp_known_orientations()` |
| Spherical radius | Configurable (default 0.5m) |

### TBP Reference Implementation

- **Sensors**: `tbp/monty/simulators/habitat/sensors.py`
  - Uses quaternion (w,x,y,z) format, default identity: (1, 0, 0, 0)
  - Position relative to HabitatAgent

- **Actions**: `tbp/monty/frameworks/actions/actions.py`
  - `LookDown`, `LookUp`: rotation_degrees parameter
  - `TurnLeft`, `TurnRight`: yaw rotation
  - `SetYaw`, `SetAgentPitch`: absolute rotations

- **Benchmarks**:
  - 3 known rotations for quick tests: [0,0,0], [0,90,0], [0,180,0]
  - 14 known orientations for full training (cube faces + corners)
  - 10 random rotations for generalization testing

## YCB Dataset Setup

**Programmatic download (via ycbust library):**

```rust
use bevy_sensor::ycb::{download_models, Subset, models_exist};

// Check if models already exist
if !models_exist("/tmp/ycb") {
    // Download representative subset (3 objects) - async
    download_models("/tmp/ycb", Subset::Representative).await?;
}

// Get paths to specific object files
let mesh = bevy_sensor::ycb::object_mesh_path("/tmp/ycb", "003_cracker_box");
let texture = bevy_sensor::ycb::object_texture_path("/tmp/ycb", "003_cracker_box");
```

**Available subsets:**
- `Subset::Representative` - 3 objects (quick testing)
- `Subset::Ten` - 10 objects (TBP benchmark subset)
- `Subset::All` - All 77 YCB objects

The `assets/ycb` symlink points to `/tmp/ycb`.

## Dependencies

```toml
bevy = { version = "0.15", default-features = false, features = [
    "bevy_asset",
    "bevy_core_pipeline",
    "bevy_pbr",
    "bevy_render",
    "bevy_scene",
    "bevy_state",
    "bevy_winit",
    "png",
    "x11",
    "tonemapping_luts",
    "ktx2",
    "zstd",
] }
bevy_obj = { version = "0.15", features = ["scene"] }
ycbust = "0.2.3"
```

### Bevy 0.15 Upgrade (December 2024)

The upgrade from Bevy 0.11 to 0.15 enabled real GPU depth buffer readback:

**Key Changes:**
- `ViewDepthTexture` replaces `ViewPrepassTextures` for depth access
- `Screenshot` entity + observer pattern replaces `ScreenshotManager`
- Component-based spawning (`Camera3d`, `PointLight`) replaces bundles
- `Mesh3d` and `MeshMaterial3d<M>` wrappers for mesh/material handles
- `SceneRoot(handle)` replaces `SceneBundle`
- MSAA must be disabled (`Msaa::Off`) for depth texture copy

**Depth Buffer Implementation:**
- Uses `ViewDepthTexture` from `bevy::render::view`
- Reverse-Z depth converted to linear meters via `reverse_z_to_linear_depth()`
- Custom render graph node (`DepthReadbackNode`) copies depth after main pass
- 256-byte row alignment for GPU buffer mapping

## Known Limitations

1. **WSL2 GPU rendering** (CRITICAL):
   - WSL2 does NOT support Vulkan window surfaces, even with WSLg display
   - `render_to_buffer()` will fail with "Invalid surface" error on WSL2
   - **Solution**: Use pre-rendered fixtures for CI/CD (see below)

2. **Software rendering (llvmpipe)**:
   - Must disable tonemapping (`Tonemapping::None`) on the camera
   - PBR shaders may fail with `gsamplerCubeArrayShadow` error
   - Xvfb may crash due to NVIDIA driver conflicts in WSL2

3. **Texture loading with bevy_obj**:
   - The bevy_obj scene loader has "limited MTL support"
   - **Workaround**: Load textures manually with `asset_server.load()`
   - MTL files with trailing spaces may cause loading failures

4. **Asset path**: When running binary from `target/release/`, assets must be in `target/release/assets/`

## Pre-rendered Fixtures for CI/CD

For environments without GPU rendering (WSL2, CI servers, Docker), use pre-rendered fixtures:

### Generating Fixtures

Run on a machine with a working display (Linux desktop with GPU):

```bash
# Generate test fixtures for CI/CD
cargo run --bin prerender -- --output-dir test_fixtures/renders

# Custom objects
cargo run --bin prerender -- --objects "003_cracker_box,005_tomato_soup_can,006_mustard_bottle"
```

This creates:
- `test_fixtures/renders/metadata.json` - Dataset configuration
- `test_fixtures/renders/{object_id}/` - Per-object renders
  - `r{N}_v{M}.png` - RGBA images
  - `r{N}_v{M}.depth` - Depth data (binary f32)
  - `index.json` - Per-object metadata

### Loading Fixtures in Tests

```rust
use bevy_sensor::fixtures::TestFixtures;

// Load pre-rendered data
let fixtures = TestFixtures::load("test_fixtures/renders")?;

// Get a specific render
let output = fixtures.get_render("003_cracker_box", 0, 0)?; // rotation 0, viewpoint 0

// Use like normal RenderOutput
let rgb = output.to_rgb_image();
let depth = output.to_depth_image();
```

### neocortx Integration with Fixtures

```rust
// In neocortx tests, prefer fixtures for CI/CD compatibility
#[cfg(test)]
mod tests {
    use bevy_sensor::fixtures::TestFixtures;

    #[test]
    fn test_sensor_with_fixtures() {
        let fixtures = TestFixtures::load("test_fixtures/renders")
            .expect("Pre-rendered fixtures required. Run: cargo run --bin prerender");

        // Test with pre-rendered data
        for (object_id, rotation_idx, viewpoint_idx, output) in fixtures.iter_renders() {
            // Process render output...
        }
    }
}
```

### WSL2 Workarounds

If you need live rendering on WSL2, these options may work (unreliable):

1. **VcXsrv on Windows**: Install VcXsrv, set `DISPLAY=172.x.x.x:0`
2. **X410 on Windows**: Commercial X server with better GPU support
3. **Native Linux VM**: Use VirtualBox/VMware with GPU passthrough

**Recommended**: Generate fixtures on a Linux machine and commit to repo for CI/CD use.

## Viewpoint Coordinate Reference

```
View  0-7:  pitch=-30° (below), yaw=0°-315° @ 45° steps, Y=-0.250
View  8-15: pitch=0°   (level), yaw=0°-315° @ 45° steps, Y=0.000
View 16-23: pitch=+30° (above), yaw=0°-315° @ 45° steps, Y=+0.250
```

## Usage from neocortx

```rust
use bevy_sensor::{SensorConfig, ObjectRotation, ViewpointConfig};
use bevy_sensor::ycb::{download_models, Subset, models_exist};

// Ensure YCB models are available
if !models_exist("/tmp/ycb") {
    download_models("/tmp/ycb", Subset::Representative).await?;
}

// Create capture config
let config = SensorConfig {
    viewpoints: ViewpointConfig {
        radius: 0.5,
        yaw_count: 8,
        pitch_angles_deg: vec![-30.0, 0.0, 30.0],
    },
    object_rotations: ObjectRotation::tbp_benchmark_rotations(),
    output_dir: "./captures".to_string(),
    filename_pattern: "ycb_{rot}_{view}.png".to_string(),
};

println!("Total captures: {}", config.total_captures()); // 72
```

## Related Projects

- **neocortx**: Rust-based Thousand Brains implementation (main project)
- **tbp.monty**: Original Python implementation by Thousand Brains Project
- **tbp.tbs_sensorimotor_intelligence**: TBP experiment configs
- **ycbust**: YCB dataset downloader (used as library dependency)

## API Parity with neocortx

bevy-sensor provides complete API parity with neocortx's `bevy_simulator` module:

| neocortx Requirement | bevy-sensor API | Status |
|---------------------|-----------------|--------|
| RGBA image data | `RenderOutput.rgba: Vec<u8>` | ✅ |
| Depth buffer (meters) | `RenderOutput.depth: Vec<f32>` | ✅ |
| Camera intrinsics | `CameraIntrinsics` struct | ✅ |
| Camera position | `RenderOutput.camera_transform` | ✅ |
| Object rotation | `RenderOutput.object_rotation` | ✅ |
| RGB image format | `to_rgb_image() → Vec<Vec<[u8; 3]>>` | ✅ |
| Depth image format | `to_depth_image() → Vec<Vec<f32>>` | ✅ |
| TBP viewpoints (24) | `ViewpointConfig::default()` | ✅ |
| TBP benchmark rotations (3) | `ObjectRotation::tbp_benchmark_rotations()` | ✅ |
| TBP full rotations (14) | `ObjectRotation::tbp_known_orientations()` | ✅ |
| YCB model download | `ycb::download_models()` | ✅ |
| Resolution config | `RenderConfig` (64×64, 256×256, 512×512) | ✅ |

**neocortx integration example:**

```rust
use bevy_sensor::{render_to_buffer, RenderConfig, ViewpointConfig, ObjectRotation};

// Render and convert to neocortx formats
let output = render_to_buffer(object_dir, &viewpoint, &rotation, &config)?;
let rgb_image: Vec<Vec<[u8; 3]>> = output.to_rgb_image();    // VisionObservation.image
let depth_image: Vec<Vec<f32>> = output.to_depth_image();    // For surface normal computation
let intrinsics = output.intrinsics;                          // VisionIntrinsics
```

## Resources

- [TBP Benchmark Experiments](https://thousandbrainsproject.readme.io/docs/benchmark-experiments)
- [TBP GitHub](https://github.com/thousandbrainsproject/tbp.monty)
- [YCB Object Dataset](https://www.ycbbenchmarks.com/)