bevy-sensor 0.3.0

Bevy library for capturing multi-view images of 3D OBJ models (YCB dataset) for sensor simulation
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

# bevy-sensor

A Bevy library and CLI for capturing multi-view images of 3D OBJ models, designed for [Thousand Brains Project](https://github.com/thousandbrainsproject/tbp.monty) compatible sensor simulation.

## Features

- Multi-viewpoint capture using spherical coordinates
- Object rotation support matching TBP benchmark formats
- Library API for integration with neocortx
- Programmatic YCB model downloads via [ycbust](https://crates.io/crates/ycbust) (v0.2.4)
- Pre-configured TBP benchmark and training configurations

## Requirements

- Rust 1.70+
- Bevy 0.15
- GPU with Vulkan support (required for headless rendering on WSL2/Linux)

## Setup

### Getting YCB Models

**Programmatically (recommended):**

```rust
use bevy_sensor::ycb::{download_models, Subset};

// Download representative subset (3 objects)
download_models("/tmp/ycb", Subset::Representative).await?;

// Or download 10 objects for TBP benchmark testing
download_models("/tmp/ycb", Subset::Ten).await?;
```

The `assets/ycb` symlink points to `/tmp/ycb`.

## Usage

### CLI

```bash
cargo run --release
```

Default configuration captures 72 images (3 rotations × 24 viewpoints) matching TBP benchmark format.

### Library

```rust
use bevy_sensor::{SensorConfig, ObjectRotation, ViewpointConfig};

// TBP benchmark: 3 rotations × 24 viewpoints = 72 captures
let config = SensorConfig::tbp_benchmark();

// Full training: 14 rotations × 24 viewpoints = 336 captures
let config = SensorConfig::tbp_full_training();

// Custom configuration
let config = SensorConfig {
    viewpoints: ViewpointConfig {
        radius: 0.5,
        yaw_count: 8,
        pitch_angles_deg: vec![-30.0, 0.0, 30.0],
    },
    object_rotations: ObjectRotation::tbp_benchmark_rotations(),
    output_dir: "./captures".to_string(),
    filename_pattern: "capture_{rot}_{view}.png".to_string(),
};
```

### YCB Utilities

```rust
use bevy_sensor::ycb::{models_exist, object_mesh_path, object_texture_path, REPRESENTATIVE_OBJECTS};

// Check if models are downloaded
if !models_exist("/tmp/ycb") {
    download_models("/tmp/ycb", Subset::Representative).await?;
}

// Get paths to object files
let mesh = object_mesh_path("/tmp/ycb", "003_cracker_box");
let texture = object_texture_path("/tmp/ycb", "003_cracker_box");

// List available objects
for obj in REPRESENTATIVE_OBJECTS {
    println!("{}", obj);
}
```

## TBP Alignment

| TBP Benchmark | bevy-sensor |
|---------------|-------------|
| 3 known rotations `[0,0,0], [0,90,0], [0,180,0]` | `ObjectRotation::tbp_benchmark_rotations()` |
| 14 known orientations (faces + corners) | `ObjectRotation::tbp_known_orientations()` |
| Distant agent look up/down | Pitch angles: -30°, 0°, +30° |
| Turn left/right | 8 yaw positions @ 45° intervals |

## Headless Rendering

For headless GPU rendering on WSL2/Linux, use the Vulkan backend:

```bash
WGPU_BACKEND=vulkan cargo run --release
```

The library uses true headless rendering with `RenderTarget::Image` - no display or window surface required.

## Output

Files are saved as `capture_{rotation}_{viewpoint}.png`:
- Rotation 0: identity `[0,0,0]`
- Rotation 1: 90° yaw `[0,90,0]`
- Rotation 2: 180° yaw `[0,180,0]`
- Viewpoints 0-23: spherical positions around object

## License

MIT