Crate bevy_sensor 

bevy-sensor: Multi-view rendering for YCB object dataset

This library provides Bevy-based rendering of 3D objects from multiple viewpoints, designed to match TBP (Thousand Brains Project) habitat sensor conventions for use in neocortx sensorimotor learning experiments.

Headless Rendering (NEW)

Render directly to memory buffers for use in sensorimotor learning:

use bevy_sensor::{render_to_buffer, RenderConfig, ViewpointConfig, ObjectRotation};
use std::path::Path;

let config = RenderConfig::tbp_default(); // 64x64, RGBD
let viewpoint = bevy_sensor::generate_viewpoints(&ViewpointConfig::default())[0];
let rotation = ObjectRotation::identity();

let output = render_to_buffer(
    Path::new("/tmp/ycb/003_cracker_box"),
    &viewpoint,
    &rotation,
    &config,
)?;

// output.rgba: Vec<u8> - RGBA pixels (64*64*4 bytes)
// output.depth: Vec<f32> - Depth values (64*64 floats)

File-based Capture (Legacy)

use bevy_sensor::{SensorConfig, ViewpointConfig, ObjectRotation};

let config = SensorConfig {
    viewpoints: ViewpointConfig::default(),
    object_rotations: ObjectRotation::tbp_benchmark_rotations(),
    ..Default::default()
};

YCB Dataset

Download YCB models programmatically:

use bevy_sensor::ycb::{download_models, Subset};

// Download representative subset (3 objects)
download_models("/tmp/ycb", Subset::Representative).await?;

Re-exports

pub use batch::BatchRenderConfig;

pub use batch::BatchRenderError;

pub use batch::BatchRenderOutput;

pub use batch::BatchRenderRequest;

pub use batch::BatchRenderer;

pub use batch::BatchState;

pub use batch::RenderStatus;

pub use ycbust;

Modules

backend

WebGPU and cross-platform backend support for rendering.

batch

Batch rendering API for multiple viewpoints and objects.

benchmark

Benchmark helpers for renderer throughput artifacts.

cache

Model caching system for efficient multi-viewpoint rendering.

fixtures

Test fixtures for pre-rendered YCB images

ycb

YCB dataset utilities

Structs

CameraIntrinsics

Camera intrinsic parameters for 3D reconstruction.

CaptureCamera

Marker component for the capture camera

CaptureTarget

Marker component for the target object being captured

CenterHitMiss

Center-hit miss with enough metadata to reproduce the bad viewpoint.

CenterHitRotationReport

Center-hit validation result for a single object rotation.

CenterHitValidationReport

Structured center-hit validation report for one object.

DownloadOptions

Options for downloading YCB objects.

LightingConfig

Lighting configuration for rendering.

MeshBounds

Axis-aligned mesh bounds in object-local coordinates.

MeshBoundsMetadata

Serializable mesh-bounds metadata for reports and manifests.

ObjectRotation

Object rotation in Euler angles (degrees), matching TBP benchmark format. Format: [pitch, yaw, roll] or [x, y, z] rotation.

PersistentRenderer

Per-step persistent renderer for feedback loops. See the module docs in render::PersistentRenderer for lifetime, thread-affinity, and object/config-invariance guarantees. Built for the surface-policy use case in neocortx where a fixed object is rendered from a moving camera many times per episode (issue #65). Persistent per-step renderer. Loads the scene once at new() and renders one frame per render() call by mutating the camera transform and scene root rotation in-place. Built for surface-policy feedback loops where the object stays fixed for the duration of an episode and the camera moves every step. See issue #65.

Quat

A quaternion representing an orientation.

RenderConfig

Configuration for headless rendering.

RenderHealth

Cheap diagnostics derived from a rendered depth buffer.

RenderOutput

Output from headless rendering containing RGBA and depth data.

RenderSession

Persistent batch render session. See the module docs in render::RenderSession for lifetime, thread-affinity, and config-invariance guarantees. Persistent batch render session. Keeps a Bevy App (and its RenderDevice plus PSO cache) alive across multiple render() calls, amortizing per-episode cold-init cost.

SensorConfig

Full sensor configuration for capture sessions

TargetedViewpoints

Generated viewpoints plus the target metadata used to create them.

Transform

Describe the position of an entity. If the entity has a parent, the position is relative to its parent position.

Vec3

A 3-dimensional vector.

ViewpointConfig

Configuration for viewpoint generation matching TBP habitat sensor behavior. Uses spherical coordinates to capture objects from multiple elevations.

Enums

RenderError

Errors that can occur during rendering and file operations.

TargetingPolicy

Render-target selection policy for TBP/YCB camera orbits.

YcbSubset

Subset of objects to download.

Constants

GOOGLE_16K_MESH_RELATIVE

Relative path from a per-object directory to the google_16k mesh file.

RENDERER_POLICY_VERSION

Stable renderer/targeting-policy version for cache manifests.

REPRESENTATIVE_OBJECTS

Representative subset of 3 commonly used objects.

TBP_SIMILAR_OBJECTS

TBP similar 10-object benchmark set (harder — similar geometry).

TBP_STANDARD_OBJECTS

TBP standard 10-object benchmark set (distinct objects).

Functions

create_batch_renderer

Create a new batch renderer helper for multi-viewpoint workflows.

generate_object_centered_viewpoints

Generate TBP viewpoint transforms around a rotated object mesh center.

generate_targeted_viewpoints

Generate viewpoints for a requested targeting policy.

generate_viewpoints

Generate camera viewpoints using spherical coordinates.

generate_viewpoints_around_target

Generate camera viewpoints around an explicit target point.

generate_ycb_object_centered_viewpoints

Generate object-centered TBP viewpoints for a YCB object directory.

initialize

Initialize bevy-sensor rendering backend configuration.

load_mesh_bounds

Load axis-aligned bounds from an OBJ mesh.

load_ycb_mesh_bounds

Load bounds for a YCB object directory using the standard google_16k mesh.

queue_render_request

Queue a render request for batch processing.

render_all_viewpoints

Render all viewpoints and rotations for a YCB object.

render_batch

Render multiple requests in batch (convenience function).

render_next_in_batch

Process and execute the next render in the batch queue.

render_to_buffer

Render a YCB object to an in-memory buffer.

render_to_buffer_cached

Render with model caching support for efficient multi-viewpoint rendering.

render_to_buffer_cached_with_object_transform

Render using the model cache with explicit object translation and scale.

render_to_buffer_with_object_transform

Render a YCB object with an explicit object translation and scale.

render_to_buffer_with_target

Render a YCB object and attach the target metadata used for the camera pose.

render_to_buffer_with_target_and_object_transform

Render a YCB object with explicit object transform and target metadata.

render_to_files

Render directly to files (for subprocess mode).

render_to_files_with_object_transform

Render directly to files with explicit object translation and scale.

rotated_mesh_center

Rotate an object-local mesh center into the rendered world frame.

validate_center_hits

Validate that each rotation has at least one viewpoint whose center pixel lands on foreground before the render far plane.