Crate datafrost

Data format and acceleration structure management

datafrost is a data-oriented resource management and scheduling library. It implements a graphics API-inspired interface that allows one to cleanly and efficiently:

• Create primary data objects, and define “derived” datatypes whose contents are generated from the primary format.
• Track how a primary object changes and automatically update the affected parts of the derived formats.
• Schedule commands to asynchronously and concurrently read or modify data objects.
  • datafrost guarantees optimal scheduling by building a directed acyclic graph to represent pending operations.
  • Multiple commands which reference different data, or immutably reference the same data, will execute in parallel.
  • Commands which mutably access the same data run in sequence, without the possibility of data races.
• Map the contents of data objects and read their results on the main thread.

Usage

The following is an abridged example of how to use datafrost. The full code may be found in the examples folder. To begin, we define the data formats that our code will use:

use datafrost::*;
use std::ops::*;

/// First, we define a general "kind" of data that our program will use.
/// In this case, let's imagine that we want to efficiently deal with
/// arrays of numbers.
pub struct NumberArray;

/// Defines the layout of an array of numbers.
pub struct NumberArrayDescriptor {
    /// The length of the array.
    pub len: usize
}

impl Kind for NumberArray { .. }

/// Next, we define the primary data format that we would like
/// to use and modify - an array of specifically `u32`s.
pub struct PrimaryArray(Vec<u32>);

impl Format for PrimaryArray { .. }

/// Now, let's imagine that we want to efficiently maintain an
/// acceleration structure containing all of the numbers in
/// the array, but doubled. So, we define the format.
pub struct DoubledArray(Vec<u32>);

impl Format for DoubledArray { .. }

/// Our goal is for `datafrost` to automatically update the doubled
/// array whenever the primary array changes. Thus, we implement
/// a way for it do so.
pub struct DoublePrimaryArray;

impl DerivedDescriptor<PrimaryArray> for DoublePrimaryArray {
    type Format = DoubledArray;

    fn update(&self, data: &mut DoubledArray, parent: &PrimaryArray, usages: &[&Range<usize>]) {
        // Loop over all ranges of the array that have changed, and
        // for each value in the range, recompute the data.
        for range in usages.iter().copied() {
            for i in range.clone() {
                data.0[i] = 2 * parent.0[i];
            }
        }
    }
}

Now that our data and its derived formats are defined, we can create instances of it and schedule commands to act upon the data:

// Create a new context.
let ctx = DataFrostContext::new(ContextDescriptor {
    label: Some("my context")
});

// Allocate a new primary array object, which has a doubled
// array as a derived format.
let data = ctx.allocate::<PrimaryArray>(AllocationDescriptor {
    descriptor: NumberArrayDescriptor { len: 7 },
    label: Some("my data"),
    derived_formats: &[Derived::new(DoublePrimaryArray)]
});

// Create a command buffer to record operations to execute
// on our data.
let mut command_buffer = CommandBuffer::new(CommandBufferDescriptor { label: Some("my command buffer") });

// Schedule a command to fill the primary number array with some data.
let view = data.view::<PrimaryArray>();
let view_clone = view.clone();
command_buffer.schedule(CommandDescriptor {
    label: Some("fill array"),
    views: &[&view.as_mut(4..6)],
    command: move |ctx| ctx.get_mut(&view_clone).0[4..6].fill(33)
});

// Schedule a command to map the contents of the derived acceleration structure
// so that we may view them synchronously.
let derived = command_buffer.map(&data.view::<DoubledArray>().as_const());

// Submit the buffer for processing.
ctx.submit(Some(command_buffer));

// The doubled acceleration structure automatically contains the
// correct, up-to-date data!
assert_eq!(&[0, 0, 0, 0, 66, 66, 0], &ctx.get(&derived).0[..]);

Structs

• AllocationDescriptor
  Determines how a new data object will be allocated.
• CommandBuffer
  Records a list of operations that should be executed on formatted data.
• CommandBufferDescriptor
  Specifies the parameters of a new CommandBuffer.
• CommandBufferStatus
  Describes the present status of a submitted command buffer.
• CommandBufferSubmission
  Uniquely identifies a submitted command buffer, allowing one to query its completion status.
• CommandContext
  Allows for interacting with format data during command execution. When this object is dropped, a command is considered complete.
• CommandDescriptor
  Describes a command to execute.
• Const
  Marks an immutable type.
• ContextDescriptor
  Determines how a DataFrostContext should behave.
• Data
  References an object of a specific Kind. The object is backed by a single primary format and some number of derived formats.
• DataFrostContext
  Manages a collection of formatted data, coordinates the execution of commands against the data, and automatically updates derived formats using the results of those commands.
• Derived
  Marks a derived format that should be accessible and kept automatically up-to-date when the parent object changes.
• Mapped
  A handle created by calling CommandBuffer::map which allows for synchronously accessing the data of a View.
• Mut
  Marks a mutable type.
• View
  References a specified format underlying a Data instance.
• ViewDescriptor
  Declares how a command will use a view. Created by calling View::as_const or View::as_mut.
• ViewRef
  A guard which allows access to the data of a format.

Traits

• DerivedDescriptor
  Allows a format to act as a maintained “copy” of a parent by synchronizing its contents with those of the parent.
• Format
  A certain format of data.
• Kind
  Denotes a general class of formats, which all share similar data. Formats of the same kind may be derived from one another.
• Mutability
  Marks whether a type is mutable or immutable.
• ViewUsage
  A type-erased instance of View that is used to specify data usage in a CommandDescriptor.