Struct rerun::DataStore

pub struct DataStore { /* private fields */ }

Everything needed to build custom StoreSubscribers. A complete data store: covers all timelines, all entities, everything.

Debugging

DataStore provides a very thorough Display implementation that makes it manageable to know what’s going on internally. For even more information, you can set RERUN_DATA_STORE_DISPLAY_SCHEMAS=1 in your environment, which will result in additional schema information being printed out.

Implementations

impl DataStore

pub fn new( id: StoreId, cluster_key: ComponentName, config: DataStoreConfig ) -> DataStore

See Self::cluster_key for more information about the cluster key.

pub fn id(&self) -> &StoreId

pub fn insert_id_component_name() -> ComponentName

The column name used for storing insert requests’ IDs alongside the data when manipulating dataframes.

See DataStoreConfig::store_insert_ids.

pub fn generation(&self) -> StoreGeneration

Return the current StoreGeneration. This can be used to determine whether the database has been modified since the last time it was queried.

pub fn cluster_key(&self) -> ComponentName

See Self::cluster_key for more information about the cluster key.

pub fn config(&self) -> &DataStoreConfig

See DataStoreConfig for more information about configuration.

pub fn lookup_datatype(&self, component: &ComponentName) -> Option<&DataType>

Lookup the arrow DataType of a re_types_core::Component in the internal DataTypeRegistry.

pub fn oldest_time_per_timeline(&self) -> BTreeMap<Timeline, TimeInt>

The oldest time for which we have any data.

Ignores timeless data.

Useful to call after a gc.

pub fn iter_indices(&self) -> impl ExactSizeIterator

Returns a read-only iterator over the raw indexed tables.

Do not use this to try and assert the internal state of the datastore.

impl DataStore

pub fn to_rows(&self) -> Result<Vec<DataRow>, DataReadError>

Serializes the entire datastore into one big sorted list of DataRow.

Individual re_log_types::DataRows that were split apart due to bucketing are merged back together.

Beware: this is extremely costly, don’t use this in hot paths.

pub fn to_data_table(&self) -> Result<DataTable, DataReadError>

Serializes the entire datastore into one big sorted DataTable.

Individual re_log_types::DataRows that were split apart due to bucketing are merged back together.

Beware: this is extremely costly, don’t use this in hot paths.

pub fn to_data_tables( &self, time_filter: Option<(Timeline, TimeRange)> ) -> impl Iterator<Item = DataTable>

Serializes the entire datastore into an iterator of DataTables, where each table corresponds 1-to-1 to an internal bucket.

impl DataStore

pub fn gc( &mut self, options: &GarbageCollectionOptions ) -> (Vec<StoreEvent>, DataStoreStats)

Triggers a garbage collection according to the desired target.

Garbage collection’s performance is bounded by the number of buckets in each table (for each RowId, we have to find the corresponding bucket, which is roughly O(log(n))) as well as the number of rows in each of those buckets (for each RowId, we have to sort the corresponding bucket (roughly O(n*log(n))) and then find the corresponding row (roughly O(log(n)). The size of the data itself has no impact on performance.

Returns the list of RowIds that were purged from the store.

Semantics

Garbage collection works on a row-level basis and is driven by RowId order, i.e. the order defined by the clients’ wall-clocks, allowing it to drop data across the different timelines in a fair, deterministic manner. Similarly, out-of-order data is supported out of the box.

The garbage collector doesn’t deallocate data in and of itself: all it does is drop the store’s internal references to that data (the DataCells), which will be deallocated once their reference count reaches 0.

Limitations

The garbage collector has limited support for latest-at semantics. The configuration option: GarbageCollectionOptions::protect_latest will protect the N latest values of each component on each timeline. The only practical guarantee this gives is that a latest-at query with a value of max-int will be unchanged. However, latest-at queries from other arbitrary points in time may provide different results pre- and post- GC.

impl DataStore

pub fn query_latest_component_with_log_level<C>( &self, entity_path: &EntityPath, query: &LatestAtQuery, level: Level ) -> Option<VersionedComponent<C>>

where C: Component,

Get the latest value for a given re_types_core::Component, as well as the associated data time and RowId.

This assumes that the row we get from the store only contains a single instance for this component; it will generate a log message of level otherwise.

This should only be used for “mono-components” such as Transform and Tensor.

This is a best-effort helper, it will merely log messages on failure.

pub fn query_latest_component<C>( &self, entity_path: &EntityPath, query: &LatestAtQuery ) -> Option<VersionedComponent<C>>

where C: Component,

Get the latest value for a given re_types_core::Component, as well as the associated data time and RowId.

This assumes that the row we get from the store only contains a single instance for this component; it will log a warning otherwise.

This should only be used for “mono-components” such as Transform and Tensor.

This is a best-effort helper, it will merely log errors on failure.

pub fn query_latest_component_quiet<C>( &self, entity_path: &EntityPath, query: &LatestAtQuery ) -> Option<VersionedComponent<C>>

where C: Component,

Get the latest value for a given re_types_core::Component, as well as the associated data time and RowId.

This assumes that the row we get from the store only contains a single instance for this component; it will return None and log a debug message otherwise.

This should only be used for “mono-components” such as Transform and Tensor.

This is a best-effort helper, it will merely logs debug messages on failure.

pub fn query_latest_component_at_closest_ancestor<C>( &self, entity_path: &EntityPath, query: &LatestAtQuery ) -> Option<(EntityPath, VersionedComponent<C>)>

where C: Component,

Call Self::query_latest_component at the given path, walking up the hierarchy until an instance is found.

pub fn query_timeless_component<C>( &self, entity_path: &EntityPath ) -> Option<VersionedComponent<C>>

where C: Component,

Get the latest value for a given re_types_core::Component and the associated RowId, assuming it is timeless.

This assumes that the row we get from the store only contains a single instance for this component; it will log a warning otherwise.

This should only be used for “mono-components” such as Transform and Tensor.

This is a best-effort helper, it will merely log errors on failure.

pub fn query_timeless_component_quiet<C>( &self, entity_path: &EntityPath ) -> Option<VersionedComponent<C>>

where C: Component,

Get the latest value for a given re_types_core::Component and the associated RowId, assuming it is timeless.

This assumes that the row we get from the store only contains a single instance for this component; it will return None and log a debug message otherwise.

This should only be used for “mono-components” such as Transform and Tensor.

This is a best-effort helper, it will merely log debug on failure.

impl DataStore

pub fn insert_component<'a, C>( &mut self, entity_path: &EntityPath, timepoint: &TimePoint, component: C )

where C: Component + Clone + 'a, Cow<'a, C>: From<C>,

Stores a single value for a given re_types_core::Component.

This is a best-effort helper, it will merely log errors on failure.

pub fn insert_empty_component( &mut self, entity_path: &EntityPath, timepoint: &TimePoint, component: ComponentName )

Stores a single empty value for a given re_types_core::ComponentName.

This is a best-effort helper, it will merely log errors on failure.

impl DataStore

pub fn all_components( &self, timeline: &Timeline, ent_path: &EntityPath ) -> Option<Vec<ComponentName>>

Retrieve all the ComponentNames that have been written to for a given EntityPath on a specific Timeline.

Temporal semantics

In addition to the temporal results, this also includes all ComponentNames present in the timeless tables for this entity.

pub fn entity_has_component( &self, timeline: &Timeline, ent_path: &EntityPath, component: &ComponentName ) -> bool

Check whether a given entity has a specific ComponentName on the specified timeline.

Temporal semantics

In addition to the temporal results, this also checks whether the ComponentName is present in timeless table.

pub fn entity_min_time( &self, timeline: &Timeline, ent_path: &EntityPath ) -> Option<TimeInt>

Find the earliest time at which something was logged for a given entity on the specified timeline.

Temporal semantics

Only considers temporal results—timeless data is ignored.

pub fn latest_at<const N: usize>( &self, query: &LatestAtQuery, ent_path: &EntityPath, primary: ComponentName, components: &[ComponentName; N] ) -> Option<(Option<TimeInt>, RowId, [Option<DataCell>; N])>

Queries the datastore for the cells of the specified components, as seen from the point of view of the so-called primary component.

Returns an array of DataCells (as well as the associated data time and RowId) on success. Success is defined by one thing and one thing only: whether a cell could be found for the primary component. The presence or absence of secondary components has no effect on the success criteria.

Temporal semantics

Temporal indices take precedence, then timeless tables are queried to fill the holes left by missing temporal data.

pub fn range<'a, const N: usize>( &'a self, query: &RangeQuery, ent_path: &EntityPath, components: [ComponentName; N] ) -> impl Iterator<Item = (Option<TimeInt>, RowId, [Option<DataCell>; N])> + 'a

Iterates the datastore in order to return the cells of the specified components, as seen from the point of view of the so-called primary component, for the given time range.

For each and every relevant row that is found, the returned iterator will yield an array that is filled with the cells of each and every component in components, or None if said component is not available in that row. A row is considered iff it contains data for the primary component.

This method cannot fail! If there’s no data to return, an empty iterator is returned.

⚠ Contrary to latest-at queries, range queries can and will yield multiple rows for a single timestamp if that timestamp happens to hold multiple entries for the primary component. On the contrary, they won’t yield any rows that don’t contain an actual value for the primary component, even if said rows do contain a value for one the secondaries!

Temporal semantics

Yields the contents of the temporal indices. Iff the query’s time range starts at TimeInt::MIN, this will yield the contents of the timeless tables before anything else.

When yielding timeless entries, the associated time will be None.

pub fn get_msg_metadata( &self, row_id: &RowId ) -> Option<&(TimePoint, EntityPathHash)>

pub fn sort_indices_if_needed(&self)

Sort all unsorted indices in the store.

impl DataStore

pub fn sanity_check(&self) -> Result<(), SanityError>

Runs the sanity check suite for the entire datastore.

Returns an error if anything looks wrong.

impl DataStore

pub fn num_timeless_rows(&self) -> u64

Returns the number of timeless index rows stored across this entire store, i.e. the sum of the number of rows across all of its timeless indexed tables.

pub fn timeless_size_bytes(&self) -> u64

Returns the size of the timeless index data stored across this entire store, i.e. the sum of the size of the data stored across all of its timeless indexed tables, in bytes.

pub fn num_temporal_rows(&self) -> u64

Returns the number of temporal index rows stored across this entire store, i.e. the sum of the number of rows across all of its temporal indexed tables.

pub fn temporal_size_bytes(&self) -> u64

Returns the size of the temporal index data stored across this entire store, i.e. the sum of the size of the data stored across all of its temporal indexed tables, in bytes.

pub fn num_temporal_buckets(&self) -> u64

Returns the number of temporal indexed buckets stored across this entire store.

pub fn entity_stats( &self, timeline: Timeline, entity_path_hash: EntityPathHash ) -> EntityStats

Stats for a specific entity path on a specific timeline

impl DataStore

pub fn register_subscriber( subscriber: Box<dyn StoreSubscriber> ) -> StoreSubscriberHandle

Registers a StoreSubscriber so it gets automatically notified when data gets added and/or removed to/from a DataStore.

Refer to StoreEvent’s documentation for more information about these events.

Scope

Registered StoreSubscribers are global scope: they get notified of all events from all existing DataStores, including DataStores created after the subscriber was registered.

Use StoreEvent::store_id to identify the source of an event.

Late registration

Subscribers must be registered before a store gets created to guarantee that no events were missed.

StoreEvent::event_id can be used to identify missing events.

Ordering

The order in which registered subscribers are notified is undefined and will likely become concurrent in the future.

If you need a specific order across multiple subscribers, embed them into an orchestrating subscriber.

pub fn with_subscriber<V, T, F>(_: StoreSubscriberHandle, f: F) -> Option<T>

where V: StoreSubscriber, F: FnMut(&V) -> T,

Passes a reference to the downcasted subscriber to the given FnMut callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_once<V, T, F>( _: StoreSubscriberHandle, f: F ) -> Option<T>

where V: StoreSubscriber, F: FnOnce(&V) -> T,

Passes a reference to the downcasted subscriber to the given FnOnce callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_mut<V, T, F>(_: StoreSubscriberHandle, f: F) -> Option<T>

where V: StoreSubscriber, F: FnMut(&mut V) -> T,

Passes a mutable reference to the downcasted subscriber to the given callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

impl DataStore

pub fn insert_row(&mut self, row: &DataRow) -> Result<StoreEvent, WriteError>

Inserts a DataRow’s worth of components into the datastore.

If the bundle doesn’t carry a payload for the cluster key, one will be auto-generated based on the length of the components in the payload, in the form of an array of monotonically increasing u64s going from 0 to N-1.

Trait Implementations

impl Clone for DataStore

fn clone(&self) -> DataStore

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Display for DataStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.