Module pasture_core::containers

source ·
Expand description

Defines traits for the different types of buffers that pasture supports, as well as some common implementations.

The buffer hierarchy in pasture

At its core, pasture distinguishes between two types of buffer properties:

  1. Who owns the memory of the buffer?
  2. How is the point data layed out in memory?

For the first property, there are three options:

  1. Memory is owned by the buffer itself (like in a Vec<T>)
  2. Memory is borrowed mutably (like in a &mut [T])
  3. Memory is borrowed immutably (like in a &[T])

For the second property, pasture knowns three different types of memory layouts:

  1. Unknown memory layout: No guarantees about the memory layout, reading and writing point data works exclusively by value
  2. Interleaved memory layout: All attributes for a single point are stored together in memory. This is similar to storing a Vec<T> where T is some struct whose members are the point attributes (like POSITION_3D, CLASSIFICATION etc.)
  3. Columnar memory layout: Data for the same attribute of multiple points is stored together in memory. This is sometimes called a ‘struct-of-arrays’ memory layout and is similar to how a column-oriented database stores its records.

Based on these two properties, pasture defines the following set of traits:

Memory ownership traits

Each buffer for point data (simply referred to as a ‘point buffer’ from here on) has to implement at least one of the memory ownership traits BorrowedBuffer, BorrowedMutBuffer, and OwningBuffer.
These correspond to the three ways of ownership of the buffer memory and form a hierarchy, where BorrowedMutBuffer implies BorrowedBuffer, and OwningBuffer implies BorrowedMutBuffer.

Memory layout traits

Optionally, a point buffer can implement one or more of the memory layout traits InterleavedBuffer, InterleavedBufferMut, ColumnarBuffer and ColumnarBufferMut. The distinction between immutable and mutable memory layout is important for slicing, which is explained in a later section.

If the buffer type stores point data in an interleaved layout, InterleavedBuffer can be implemented, which allows accessing point data by borrow (or mutable borrow through InterleavedBufferMut), either for a single point or a range of points.

If the buffer type stores point data in a columnar layout, ColumnarBuffer can be implemented, which allows accessing attribute data by borrow (or mutable borrow through ColumnarBufferMut), either for a single point or a range of points.

To illustrate this, here is an example. Given the following point type:

use nalgebra::Vector3;
// (skipped necessary derives for brevity)
struct Point {
   position: Vector3<f64>,
   classification: u8,
}

An interleaved buffer allows accessing the point data as a &[Point], whereas a columnar buffer allows accessing e.g. all positions as a &[Vector3<f64>]. In practice, the point buffer traits work with untyped data, so they will always return &[u8] or &mut [u8] instead of strongly typed values!

Slicing buffers

Just as with a Rust Vec<T>, pasture point buffers can support slicing through the SliceBuffer and SliceBufferMut traits. Given some buffer type T, calling slice(range) on the buffer will yield an immutable slice to the point data of that buffer. The slice keeps its memory layout, so if T implements InterleavedBuffer, the slice will also implement InterleavedBuffer (and conversely for all the other memory layout traits). Due to limitations of the std::ops::Index trait in Rust, a separate trait is required for slicing point buffers, meaning you cannot use the [] operator for slicing and instead have to call slice() or slice_mut() explicitly.

Raw vs. typed memory

Since the pasture point buffers store dynamically typed data (i.e. point data whose attributes are only known at runtime), the API of all the buffer traits works with byte slices (&[u8] and &mut [u8]) instead of strongly typed data. Whenever possible and viable (due to performance reasons), runtime type checking is performed using the PointLayout, which is part of every point buffer (it is mandated by the BorrowedBuffer trait). All methods that perform no type checking are marked unsafe. See their documentation for information about invariants that must hold when calling these functions.

Depending on your application, you might be able to work almost exclusively with strongly typed point data. In this case, all point buffers provide view and view_attribute methods (with mutable variants) that allow access to the point buffer through a strongly typed interface. See the buffer_views module for more information on buffer views.

Specific buffer types

Currently, pasture provides three specific buffer implementations:

  • VectorBuffer, an owning, interleaved point buffer using a Vec<u8> as its underlying storage
  • HashMapBuffer, an owning, columnar point buffer using a HashMap<PointAttributeDefinition, Vec<u8>> as its underlying storage
  • ExternalMemoryBuffer, a non-owning (though potentially mutable) interleaved point buffer which uses an arbitrary external memory resource for its underlying storage

Structs

  • AttributeIteratorByMut
    Like AttributeIteratorByRef, but returns attribute data by mutable reference, allowing mutation of the attribute data in-place. Can only be constructed from a buffer that implements ColumnarBufferMut
  • AttributeIteratorByRef
    Like AttributeIteratorByValue, but returns attribute data by immutable reference. Can only be constructed from a buffer that implements ColumnarBuffer
  • AttributeIteratorByValue
    An iterator over strongly typed attribute data in a point buffer. Returns attribute data by value and makes assumptions about the memory layout of the underlying buffer
  • AttributeView
    A strongly typed view over attribute data of a point buffer. This allows accessing the data for a specific attribute of a PointType using the strong type T instead of as raw memory (i.e. &[u8]). This type makes no assumptions about the memory layout of the underlying buffer, so it only provides access to the attribute data by value. Just as with the PointView type, you cannot create instances of AttributeView directly. Instead, use the BorrowedBuffer::view_attribute function and its variations, which perform the necessary type checking.
  • AttributeViewConverting
    A view over a strongly typed point attribute that supports type conversion. This means that the PointAttributeDataType of the attribute does not have to match the type T that this view returns. For an explanation on how attribute type conversion works in pasture, see the conversion module
  • AttributeViewConvertingIterator
    An iterator that performs attribute value conversion on the fly. This allows iterating over an attribute that has internal datatype U as if it had datatype T
  • AttributeViewMut
    Like AttributeView, but provides mutable access to the attribute data
  • BufferSlice
    An immutable slice to a point buffer. In terms of memory layout, the slice will have the same capabilities as the underlying buffer, i.e. if T implements InterleavedBuffer, so does this slice, and similar for the other memory layout traits.
  • BufferSliceColumnar
    A buffer slice for a columnar buffer
  • BufferSliceColumnarMut
    A mutable buffer slice for a columnar buffer
  • BufferSliceInterleaved
    A buffer slice for an interleaved buffer
  • BufferSliceInterleavedMut
    A mutable buffer slice for an interleaved buffer
  • BufferSliceMut
    A mutable slice to a point buffer. Works like BufferSlice, but allows mutable access to the underlying buffer. This type conditionally implements the InterleavedBufferMut and ColumnarBufferMut traits if T implements them
  • ExternalMemoryBuffer
    A point buffer that stores point data in interleaved memory layout in an externally borrowed memory resource. This can be any type that is convertible to a &[u8]. If T also is convertible to a &mut [u8], this buffer also implements [BorrowedBufferMut]
  • HashMapBuffer
    A point buffer that stores point data in columnar memory layout, using a HashMap<PointAttributeDefinition, Vec<u8>> as its underlying storage
  • HashMapBufferAttributePusher
    Helper struct to push point data into a HashMapBuffer attribute by attribute. This allows constructing a point buffer from multiple ranges of attribute data, since regular point buffers do not allow pushing just a single attribute into the buffer, as buffers always have to store complete points (even with columnar memory layout)
  • PointIteratorByMut
    Iterator over strongly typed points by mutable borrow
  • PointIteratorByRef
    Iterator over strongly typed points by immutable borrow
  • PointIteratorByValue
    Iterator over strongly typed points by value
  • PointView
    A strongly typed view over the point data of a buffer. This allows accessing the point data in the buffer using type T instead of only through raw memory (i.e. as &[u8]). This type makes no assumptions about the memory layout of the underlying buffer, so it only provides access to the point data by value. The PointView supports no type conversion, so T::layout() must match the PointLayout of the buffer. You cannot create instances of PointView directly but instead have to use BorrowedBuffer::view function and its variations, which perform the necessary type checks internally!
  • PointViewMut
    Like PointView, but provides mutable access to the strongly typed point data. For buffers with unknown memory layout, this means that you have to use PointViewMut::set_at, but if the underlying buffer implements InterleavedBufferMut, you can also get a mutable borrow the a strongly typed point!
  • RawAttributeView
    A view over raw memory for a point attribute. This view can be obtained from any buffer that is either interleaved or columnar, and will be more efficient than calling get_attribute on the buffer
  • RawAttributeViewMut
    Like RawAttributeView, but for mutable data
  • UntypedPointBuffer
    An implementaion of UntypedPoint trait that has an internal buffer.
  • UntypedPointSlice
    An implementaion of UntypedPoint trait that handles an external buffer.
  • VectorBuffer
    A point buffer that uses a Vec<u8> as its underlying storage. It stores point data in interleaved memory layout and generally behaves like an untyped vector.

Traits

  • BorrowedBuffer
    Base trait for all point buffers in pasture. The only assumption this trait makes is that the underlying memory can be borrowed by the buffer. Provides point and attribute accessors by untyped value (i.e. copying into a provided &mut [u8])
  • BorrowedMutBuffer
    Trait for a point buffer that mutably borrows its memory. Compared to BorrowedBuffer, buffers that implement this trait support the following additional capabilities:
  • ColumnarBuffer
    Trait for point buffers that store their point data in columnar memory layout. This allows accessing point attributes by reference
  • ColumnarBufferMut
    Trait for buffers that store point data in columnar memory layout and also borrow their memory mutably. Compared to ColumnarBuffer, this allows accessing point attributes by mutable reference!
  • InterleavedBuffer
    Trait for point buffers that store their point data in interleaved memory layout. This allows accessing point data by reference
  • InterleavedBufferMut
    Trait for buffers that store point data in interleaved memory layout and also borrow their memory mutably. Compared to InterleavedBuffer, this allows accessing point data by mutable reference!
  • MakeBufferFromLayout
    Trait for all buffers that can be default-constructed from a given PointLayout. This trait is helpful for generic code that needs to construct an generic buffer type
  • OwningBuffer
    Trait for point buffers that own their memory. Compared to [BorrowedBufferMut], buffers that implement this trait support the following additional capabilities:
  • SliceBuffer
    Trait for buffers that support slicing, similar to the builtin slice type
  • SliceBufferMut
    Trait for buffers that support mutable slicing
  • UntypedPoint
    A trait to handle points that layout can’t be known to compile time.