Module crayon::video

A stateless, layered, multithread video system with OpenGL backends.

Overview and Goals

The management of video effects has become an important topic and key feature of rendering engines. With the increasing number of effects it is not sufficient anymore to only support them, but also to integrate them into the rendering engine in a clean and extensible way.

The goal of this work and simultaneously its main contribution is to design and implement an advanced effects framework. Using this framework it should be easy for further applications to combine several small effects like texture mapping, shading and shadowing in an automated and transparent way and apply them to any 3D model. Additionally, it should be possible to integrate new effects and use the provided framework for rapid prototyping.

Multi Platform

Ideally, crayon should be able to run on macOS, windows and popular mobile-platforms. There still are a huge number of performance and feature limited devices, so this video module will always be limited by lower-end 3D APIs like OpenGL ES2.0.

Stateless Pipeline

Ordinary OpenGL application deals with stateful APIs, which is error-prone. This means whenever you change any state in the API for subsequent draw calls, this state change also affects draw calls submitted at a later point in time. Ideally, submitting a draw call with whatever state we want should not affect any of the other draw calls, even in multi-thread environments.

Modern 3D-APIs like gfx-rs, glium bundles render state and data into a few, precompiled resource objects which are combined into final render pipeline. We should follow the same philosophy.

Multi-thread

In most cases, dividing OpenGL rendering across multiple threads will not result in any performance improvement due the pipeline nature of OpenGL. What we are about to do is actually exploiting parallelism in resource preparation, and provides a set of multi-thread friendly APIs.

The most common solution is by using a double-buffer of commands. This consists of running the renderer backend in a speparate thread, where all draw calls and communication with the OpenGL API are performed. The frontend thread that runs the game logic communicates with the backend renderer via a command double-buffer.

Layered Rendering

Its important to sort video commands (generated by different threads) before submiting them to OpenGL, for the sack of both correctness and performance. For example, to draw transparent objects via blending, we need draw opaque object first, usually from front-to-back, and draw translucents from back-to-front.

The idea here is to assign a integer key to a command which is used for sorting. Depending on where those bits are stored in the integer, you can apply different sorting criteria for the same array of commands, as long as you know how the keys were built.

Resource Objects

Render state and data, which are combined into final render pipeline, are bundled into a few, precompiled resource objects in video module.

All resources types can be created instantly from data in memory, and meshes, textures can also be loaded asynchronously from the filesystem.

And the actual resource objects are usually private and opaque, you will get a Handle immediately for every resource objects you created instead of some kind of reference. Its the unique identifier for the resource, its type-safe and copyable.

When you are done with the created resource objects, its your responsiblity to delete the resource object with Handle to avoid leaks.

For these things loaded from filesystem, it could be safely shared by the Location. We keeps a use-counting internally. It will not be freed really, before all the users deletes its Handle.

Surface Object

Surface object plays as the Layer role we mentioned above, all the commands we submitted in application code is attached to a specific Surface. Commands inside Surface are sorted before submitting to underlying OpenGL.

Surface object also holds references to render target, and wraps rendering operations to it. Likes clearing, offscreen-rendering, MSAA resolve etc..

use crayon::video::prelude::*;
use crayon::math::Color;
let video = VideoSystem::headless(None).shared();

// Creates a `SurfaceParams` object.
let mut params = SurfaceParams::default();
/// Sets the attachments of internal frame-buffer. It consists of multiple color attachments
/// and a optional `Depth/DepthStencil` buffer attachment.
///
/// If none attachment is assigned, the default framebuffer generated by the system will be
/// used.
params.set_attachments(&[], None);
// Sets the clear flags for this surface and its underlying framebuffer.
params.set_clear(Color::white(), 1.0, None);

// Creates an surface with `SurfaceParams`.
let surface = video.create_surface(params).unwrap();
// Deletes the surface object.
video.delete_surface(surface);

Shader Object

Shader object is introduced to encapsulate all stateful things we need to configurate video pipeline. This would also enable us to easily change the order of draw calls and get rid of redundant state changes.

use crayon::video::prelude::*;
let video = VideoSystem::headless(None).shared();

// Declares the uniform variable layouts.
let mut uniforms = UniformVariableLayout::build()
    .with("u_ModelViewMatrix", UniformVariableType::Matrix4f)
    .with("u_MVPMatrix", UniformVariableType::Matrix4f)
    .finish();

// Declares the attributes.
let attributes = AttributeLayout::build()
     .with(Attribute::Position, 3)
     .with(Attribute::Normal, 3)
     .finish();

let mut params = ShaderParams::default();
params.attributes = attributes;
params.uniforms = uniforms;
params.state = RenderState::default();

let vs = "..".into();
let fs = "..".into();

// Create a shader with initial shaders and render state. It encapusulates all the
// informations we need to configurate graphics pipeline before real drawing.
let shader = video.create_shader(params, vs, fs).unwrap();

// Deletes shader object.
video.delete_shader(shader);

Texture Object

A texture object is a container of one or more images. It can be the source of a texture access from a Shader.

use crayon::video::prelude::*;
let video = VideoSystem::headless(None).shared();

let mut params = TextureParams::default();

// Create a texture object with optional data. You can fill it later with `update_texture`.
let texture = video.create_texture(params, None).unwrap();

// Deletes the texture object.
video.delete_texture(texture);

Compressed Texture Format

TODO: Cube texture. TODO: 3D texture.

Mesh Object

use crayon::video::prelude::*;
let video = VideoSystem::headless(None).shared();

let mut params = MeshParams::default();

// Create a mesh object with optional data. You can fill it later with `update_mesh`.
let mesh = video.create_mesh(params, None).unwrap();

// Deletes the mesh object.
video.delete_mesh(mesh);

Commands

Finally, when we finished the preparation of video resources, to make draw call:

use crayon::video::prelude::*;
use crayon::math;
let video = VideoSystem::headless(None).shared();

let mut params = SurfaceParams::default();
// ..
let surface = video.create_surface(params).unwrap();

let mut params = MeshParams::default();
// ...
let mesh = video.create_mesh(params, None).unwrap();

let mut params = ShaderParams::default();
// ...
let vs = "..".into();
let fs = "..".into();
let shader = video.create_shader(params, vs, fs).unwrap();

// Creates a draw call with specified shader and mesh object.
let mut dc = DrawCall::new(shader, mesh);
// You can set the uniform variables for this dc.
dc.set_uniform_variable("someUniform", math::Vector3::new(0.0, 0.0, 0.0));
// Commits the draw call into surface.
video.draw(surface, dc);

TODO: Batch TODO: OrderDrawBatch

Modules

assets
batch
errors
prelude

Structs

VideoFrameInfo	The information of video module during last frame.
VideoSystem	The centralized management of video sub-system.
VideoSystemShared	The multi-thread friendly parts of VideoSystem.

Constants

MAX_FRAMEBUFFER_ATTACHMENTS	Maximum number of attachments in framebuffer.
MAX_UNIFORM_TEXTURE_SLOTS	Maximum number of textures in shader.
MAX_UNIFORM_VARIABLES	Maximum number of uniform variables in shader.
MAX_VERTEX_ATTRIBUTES	Maximum number of attributes in vertex layout.

Type Definitions

MeshRegistry
TextureRegistry