Crate datafrost

Data format and acceleration structure management

datafrost is a data-oriented resource management and scheduling library. It implements a graphics API-inspired interface that allows one to cleanly and efficiently:

• Create primary data objects, and define “derived” datatypes whose contents are generated from the primary format.
• Track how a primary object changes and automatically update the affected parts of the derived formats.
• Schedule commands to asynchronously and concurrently read or modify data objects.
  • datafrost guarantees optimal scheduling by building a directed acyclic graph to represent pending operations.
  • Multiple commands which reference different data, or immutably reference the same data, will execute in parallel.
  • Commands which mutably access the same data run in sequence, without the possibility of data races.
• Map the contents of data objects and read their results on the main thread.

Usage

The following is an abridged example of how to use datafrost. The full code may be found in the examples folder. To begin, we define the data formats that our code will use:

use datafrost::*;
use std::ops::*;

/// First, we define a general "kind" of data that our program will use.
/// In this case, let's imagine that we want to efficiently deal with
/// arrays of numbers.
pub struct NumberArray;

/// Defines the layout of an array of numbers.
pub struct NumberArrayDescriptor {
    /// The length of the array.
    pub len: usize
}

impl Kind for NumberArray { .. }

/// Next, we define the primary data format that we would like
/// to use and modify - an array of specifically `u32`s.
pub struct PrimaryArray(Vec<u32>);

impl Format for PrimaryArray { .. }

/// Now, let's imagine that we want to efficiently maintain an
/// acceleration structure containing all of the numbers in
/// the array, but doubled. So, we define the format.
pub struct DoubledArray(Vec<u32>);

impl Format for DoubledArray { .. }

/// Our goal is for `datafrost` to automatically update the doubled
/// array whenever the primary array changes. Thus, we implement
/// a way for it do so.
pub struct DoublePrimaryArray;

impl DerivedDescriptor<PrimaryArray> for DoublePrimaryArray {
    type Format = DoubledArray;

    fn update(&self, data: &mut DoubledArray, parent: &PrimaryArray, usages: &[&Range<usize>]) {
        // Loop over all ranges of the array that have changed, and
        // for each value in the range, recompute the data.
        for range in usages.iter().copied() {
            for i in range.clone() {
                data.0[i] = 2 * parent.0[i];
            }
        }
    }
}

Now that our data and its derived formats are defined, we can create instances of it and schedule commands to act upon the data:

// Create a new context.
let ctx = DataFrostContext::new(ContextDescriptor {
    label: Some("my context")
});

// Allocate a new primary array object, which has a doubled
// array as a derived format.
let data = ctx.allocate::<PrimaryArray>(AllocationDescriptor {
    descriptor: NumberArrayDescriptor { len: 7 },
    label: Some("my data"),
    derived_formats: &[Derived::new(DoublePrimaryArray)]
});

// Create a command buffer to record operations to execute
// on our data.
let mut command_buffer = CommandBuffer::new(CommandBufferDescriptor { label: Some("my command buffer") });

// Schedule a command to fill the primary number array with some data.
let view = data.view::<PrimaryArray>();
let view_clone = view.clone();
command_buffer.schedule(CommandDescriptor {
    label: Some("fill array"),
    views: &[&view.as_mut(4..6)],
    command: move |ctx| ctx.get_mut(&view_clone).0[4..6].fill(33)
});

// Schedule a command to map the contents of the derived acceleration structure
// so that we may view them synchronously.
let derived = command_buffer.map(&data.view::<DoubledArray>().as_const());

// Submit the buffer for processing.
ctx.submit(Some(command_buffer));

// The doubled acceleration structure automatically contains the
// correct, up-to-date data!
assert_eq!(&[0, 0, 0, 0, 66, 66, 0], &ctx.get(&derived).0[..]);

Structs

AllocationDescriptor

Determines how a new data object will be allocated.

CommandBuffer

Records a list of operations that should be executed on formatted data.

CommandBufferDescriptor

Specifies the parameters of a new CommandBuffer.

CommandBufferStatus

Describes the present status of a submitted command buffer.

CommandBufferSubmission

Uniquely identifies a submitted command buffer, allowing one to query its completion status.

CommandContext

Allows for interacting with format data during command execution. When this object is dropped, a command is considered complete.

CommandDescriptor

Describes a command to execute.

Const

Marks an immutable type.

ContextDescriptor

Determines how a DataFrostContext should behave.

Data

References an object of a specific Kind. The object is backed by a single primary format and some number of derived formats.

DataFrostContext

Manages a collection of formatted data, coordinates the execution of commands against the data, and automatically updates derived formats using the results of those commands.

Derived

Marks a derived format that should be accessible and kept automatically up-to-date when the parent object changes.

Mapped

A handle created by calling CommandBuffer::map which allows for synchronously accessing the data of a View.

Mut

Marks a mutable type.

View

References a specified format underlying a Data instance.

ViewDescriptor

Declares how a command will use a view. Created by calling View::as_const or View::as_mut.

ViewRef

A guard which allows access to the data of a format.

Traits

DerivedDescriptor

Allows a format to act as a maintained “copy” of a parent by synchronizing its contents with those of the parent.

Format

A certain format of data.

Kind

Denotes a general class of formats, which all share similar data. Formats of the same kind may be derived from one another.

Mutability

Marks whether a type is mutable or immutable.

ViewUsage

A type-erased instance of View that is used to specify data usage in a CommandDescriptor.