Struct ChunkStore 

pub struct ChunkStore { /* private fields */ }

Everything needed to build custom ChunkStoreSubscribers. A complete chunk store: covers all timelines, all entities, everything.

The chunk store always works at the chunk level, whether it is for write & read queries or garbage collection. It is completely oblivious to individual rows.

Use the Display implementation for a detailed view of the internals.

Implementations

impl ChunkStore

pub fn compacted( &self, options: &CompactionOptions, ) -> Result<ChunkStore, ChunkStoreError>

Return a new, compacted version of this store.

Compaction merges small neighboring chunks that share the same entity path, timelines, and datatypes, up to the thresholds in the config. Large chunks may be split.

If is_start_of_gop is provided, video stream chunks are rebatched to align with GoP boundaries after compaction, and sparse is_keyframe marker chunks are emitted.

If split_size_ratio is provided, chunks are split on entry so no two archetype groups sharing a chunk differ in byte size by more than that factor.

The returned store has compaction disabled (ChunkStoreConfig::ALL_DISABLED).

pub fn finalize_compaction( self, options: &CompactionOptions, ) -> Result<ChunkStore, ChunkStoreError>

Finalize a compaction-enabled store: run up to CompactionOptions::num_extra_passes additional compaction passes (stopping early when the chunk count stops decreasing), optionally rebatch video chunks along GoP boundaries, then disable compaction on the returned store (ChunkStoreConfig::ALL_DISABLED config).

Consumes self. Assumes self was built with a compaction-enabled config (otherwise each pass is a no-op).

impl ChunkStore

pub fn row_id_descriptor(&self) -> RowIdColumnDescriptor

pub fn resolve_time_selector( &self, selector: &TimeColumnSelector, ) -> IndexColumnDescriptor

Given a TimeColumnSelector, returns the corresponding IndexColumnDescriptor.

pub fn resolve_component_selector( &self, selector: &ComponentColumnSelector, ) -> Option<ComponentColumnDescriptor>

Given a ComponentColumnSelector, returns the corresponding ComponentColumnDescriptor.

If the component is not found in the store, None is returned.

pub fn schema_for_query( &self, query: &QueryExpression, ) -> ChunkColumnDescriptors

Returns the filtered schema for the given QueryExpression.

The order of the columns is guaranteed to be in a specific order:

• first, the time columns in lexical order (frame_nr, log_time, …);
• second, the component columns in lexical order (Color, Radius, ...).

pub fn create_component_filter_from_query( query: &QueryExpression, ) -> impl Fn(&ComponentColumnDescriptor)

impl ChunkStore

pub fn drop_time_range_shallow( &mut self, timeline: &TimelineName, drop_range: AbsoluteTimeRange, reason: ChunkDeletionReason, ) -> Vec<ChunkStoreEvent>

Drop all events that are in the given range on the given timeline.

Note that matching events will be dropped from all timelines they appear on.

Chunks are shallowly removed: they can be recovered if they were originally fetched from a known RRD manifest. Static chunks are unaffected.

pub fn drop_time_range_deep( &mut self, timeline: &TimelineName, drop_range: AbsoluteTimeRange, reason: ChunkDeletionReason, ) -> Vec<ChunkStoreEvent>

Drop all events that are in the given range on the given timeline.

Note that matching events will be dropped from all timelines they appear on.

Chunks are deeply removed: they won’t be recoverable. Static chunks are unaffected.

Used to implement undo (erase the last event from the blueprint db).

impl ChunkStore

pub fn gc( &mut self, options: &GarbageCollectionOptions, ) -> (Vec<ChunkStoreEvent>, ChunkStoreStats)

Triggers a garbage collection according to the desired target.

Returns the list of physical Chunks that were purged from the store in the form of ChunkStoreEvents.

Semantics

Garbage collection works on a chunk-level basis, giving priority to those that are the furthest away from the timestamp specified in GarbageCollectionOptions::furthest_from.

If no timestamp is specified, or if not enough data could be collected during the timestamp-driven pass, then garbage collection falls back to RowId order (specifically, the smallest RowId of each respective Chunk), i.e. the order defined by the clients’ wall-clocks, allowing it to drop data across the different timelines in a fair, deterministic manner. Similarly, out-of-order data is supported out of the box.

The garbage collector doesn’t deallocate data in and of itself: all it does is drop the store’s internal references to that data (the Chunks), which will be deallocated once their reference count reaches 0.

Limitations

The garbage collector has limited support for latest-at semantics. The configuration option: GarbageCollectionOptions::protect_latest will protect the N latest values of each component on each timeline. The only practical guarantee this gives is that a latest-at query with a value of max-int will be unchanged. However, latest-at queries from other arbitrary points in time may provide different results pre- and post- GC.

pub fn remove_chunks_deep( &mut self, chunks_to_be_removed: Vec<Arc<Chunk>>, time_budget: Option<Duration>, reason: ChunkDeletionReason, ) -> Vec<ChunkStoreDiffDeletion>

Surgically removes a set of temporal ChunkIds from all physical & virtual indices.

This only makes sense to use on chunks that resulted from the compaction of other chunks. These chunks, by definition, only ever exist locally, and therefore there is never any good reason to let them linger on in our internal indices, physical or virtual, since they cannot possibly be re-fetched. Note that this only applies to compaction, not splitting, since the chunk being split never makes it into the store in the first place. For garbage collection purposes, refer to Self::remove_chunks_shallow instead.

This is orders of magnitude faster than trying to retain() on all our internal indices, when you already know where these chunks live.

pub fn remove_chunks_shallow( &mut self, chunks_to_be_removed: Vec<Arc<Chunk>>, time_budget: Option<Duration>, reason: ChunkDeletionReason, ) -> Vec<ChunkStoreDiffDeletion>

Surgically removes a set of temporal ChunkIds from all physical indices only.

This only makes sense to use with garbage collection: you want to make sure that a chunk that was garbage collected away stills lingers on in our internal virtual indices, so we can know at query time that some data was missing from local memory. into a new larger chunk to linger on in our internal indices, both physical and virtual. For compaction purposes, refer to Self::remove_chunks_deep instead.

This is orders of magnitude faster than trying to retain() on all our internal indices, when you already know where these chunks live.

impl ChunkStore

pub fn format_lineage(&self, chunk_id: &ChunkId) -> String

Formats the complete lineage tree of a chunk in a human readable fashion.

This is a debugging tool, it makes no effort whatsoever to try and be performant.

pub fn is_root_chunk(&self, chunk_id: &ChunkId) -> bool

Returns true if this is a root-level chunk.

Root-level chunks sit directly at the top of the lineage tree: they cannot be issued from either a split or a compaction. I.e. the next layer is necessarily either a reference to volatile memory, or to an RRD manifest.

pub fn find_root_chunks(&self, chunk_id: &ChunkId) -> Vec<ChunkId>

Returns the roots from which a given chunk was derived from.

Due to compaction, lineage forms a tree rather than a straight line, and therefore it is possible (and even common) for a chunk to have more than one root.

The resulting root chunks might or might not be volatile. If you only care about chunks that are still available for download, see Self::find_root_manifest_chunks.

pub fn collect_root_ids(&self, chunk_id: &ChunkId, roots: &mut Vec<ChunkId>)

See Self::find_root_chunks.

pub fn find_root_manifest_chunks(&self, chunk_id: &ChunkId) -> Vec<ChunkId>

Returns the top-level non-volatile roots of a given chunk, if any.

Due to compaction, lineage forms a tree rather than a straight line, and therefore it is possible (and even common) for a chunk to have more than one root, from possibly more than one RRD manifest.

The resulting root chunks are guaranteed to be backed by an RRD manifest (non-volatile). If you want to find all root chunks regardless of their origin, refer to Self::find_root_chunks instead.

pub fn collect_physical_descendents_of( &self, chunk_id: &ChunkId, descendents: &mut Vec<ChunkId>, )

Collects all physical chunks that descend from the given chunk in some way.

pub fn descends_from_manifest(&self, chunk: &ChunkId) -> bool

Returns true if either the specified chunk or one of its ancestors is from a manifest.

pub fn descends_from_a_split(&self, chunk_id: &ChunkId) -> bool

Returns true if either the specified chunk or one of its ancestors resulted from a split.

pub fn descends_from_a_compaction(&self, chunk_id: &ChunkId) -> bool

Returns true if either the specified chunk or one of its ancestors resulted from a compaction.

pub fn direct_lineage(&self, chunk_id: &ChunkId) -> Option<&ChunkDirectLineage>

Returns the direct lineage of a chunk.

impl ChunkStore

pub fn extract_properties(&self) -> Result<RecordBatch, ExtractPropertiesError>

Extract a one-row RecordBatch containing the properties for this chunk store.

The column names are based on the following proposals and are further sanitized to ensure compatibility with Lance datasets.

https://www.notion.so/rerunio/Canonical-column-identifier-for-dataframe-queries-206b24554b1980d98454eb989703ce2b https://www.notion.so/rerunio/Canonical-column-identifier-for-properties-215b24554b1980029ff1cc6cdfad3f76

pub fn property_entities_query_results(&self) -> Vec<(EntityPath, QueryResults)>

Run the property-entity latest-at queries used by both the pure-ChunkStore path and the split property_entities_query_results / extract_properties_from_chunks path used by lazy stores.

impl ChunkStore

pub fn all_entities( &self, ) -> HashSet<EntityPath, BuildHasherDefault<NoHashHasher<EntityPath>>>

Retrieve all EntityPaths in the store.

pub fn find_temporal_chunks_furthest_from( &self, timeline: &TimelineName, time: TimeInt, ) -> Vec<Arc<Chunk>>

Returns a vector with all the chunks in this store, sorted in descending order relative to their distance from the given (timeline, time) cursor.

pub fn all_entities_sorted(&self) -> BTreeSet<EntityPath>

Retrieve all EntityPaths in the store.

pub fn all_components( &self, ) -> HashSet<ComponentIdentifier, BuildHasherDefault<NoHashHasher<ComponentIdentifier>>>

Retrieve all ComponentIdentifiers in the store.

See also Self::all_components_sorted.

pub fn all_components_sorted(&self) -> BTreeSet<ComponentIdentifier>

Retrieve all ComponentIdentifiers in the store.

See also Self::all_components.

pub fn all_components_on_timeline( &self, timeline: Option<&TimelineName>, entity_path: &EntityPath, ) -> Option<HashSet<ComponentIdentifier, BuildHasherDefault<NoHashHasher<ComponentIdentifier>>>>

Retrieve all the ComponentIdentifiers that have been written to for a given EntityPath on the specified re_chunk::Timeline.

Static components are always included in the results.

A None timeline (a static-only query) yields only the static components.

Returns None if the entity doesn’t exist at all on this timeline.

pub fn all_components_on_timeline_sorted( &self, timeline: &TimelineName, entity_path: &EntityPath, ) -> Option<BTreeSet<ComponentIdentifier>>

Retrieve all the ComponentIdentifiers that have been written to for a given EntityPath on the specified re_chunk::Timeline.

Static components are always included in the results.

Returns None if the entity doesn’t exist at all on this timeline.

pub fn entity_has_component_on_timeline( &self, timeline: Option<&TimelineName>, entity_path: &EntityPath, component: ComponentIdentifier, ) -> bool

Check whether an entity has a static component or a temporal component on the specified timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_component( &self, entity_path: &EntityPath, component: ComponentIdentifier, ) -> bool

Check whether an entity has a static component or a temporal component on any timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_static_component( &self, entity_path: &EntityPath, component: ComponentIdentifier, ) -> bool

Check whether an entity has a specific static component.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_temporal_component( &self, entity_path: &EntityPath, component: ComponentIdentifier, ) -> bool

Check whether an entity has a temporal component on any timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_temporal_component_on_timeline( &self, timeline: &TimelineName, entity_path: &EntityPath, component: ComponentIdentifier, ) -> bool

Check whether an entity has a temporal component on a specific timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_physical_data_on_timeline( &self, timeline: &TimelineName, entity_path: &EntityPath, ) -> bool

Check whether an entity has any physical data on a specific timeline, or any static data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any indexed data, physical or virtual.

Returns true if the entity has any static or temporal chunk IDs, regardless of whether those chunks are currently loaded in memory.

An entity path can exist in the schema/entity tree but return false here if all of its chunks have been removed by garbage collection or otherwise removed.

pub fn entity_has_physical_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any physical data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_physical_static_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any static data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_physical_temporal_data( &self, entity_path: &EntityPath, ) -> bool

Check whether an entity has any temporal data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_physical_temporal_data_on_timeline( &self, entity_path: &EntityPath, timeline: &TimelineName, ) -> bool

Check whether an entity has any physical temporal data on any timeline.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_physical_temporal_data_on_timeline_for_component( &self, entity_path: &EntityPath, timeline: &TimelineName, component: &ComponentIdentifier, ) -> bool

Check whether an entity has any physical temporal data for a given component.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

See Self::entity_has_physical_temporal_data_on_timeline if you don’t care about any particular component.

pub fn entity_min_time( &self, timeline: &TimelineName, entity_path: &EntityPath, ) -> Option<TimeInt>

Find the earliest time at which something was logged for a given entity on the specified timeline.

This includes both physical & virtual chunks. Ignores static data.

pub fn entity_time_range( &self, timeline: &TimelineName, entity_path: &EntityPath, ) -> Option<AbsoluteTimeRange>

Returns the min and max times at which data was logged for an entity on a specific timeline.

This includes both physical & virtual chunks. This ignores static data.

pub fn next_time_on_timeline( &self, timeline: &TimelineName, after: TimeInt, ) -> Option<TimeInt>

Returns the next non-static time with data on the given timeline, strictly after after.

Searches physical chunks across all entities. Returns None if there is no later temporal data.

This scales linearly with the number of chunks on the timeline.

pub fn prev_time_on_timeline( &self, timeline: &TimelineName, before: TimeInt, ) -> Option<TimeInt>

Returns the previous non-static time with data on the given timeline, strictly before before.

Searches physical chunks across all entities. Returns None if there is no earlier temporal data.

This scales linearly with the number of chunks on the timeline.

pub fn time_range(&self, timeline: &TimelineName) -> Option<AbsoluteTimeRange>

Returns the min and max times at which data was logged on a specific timeline, considering all entities.

This includes both physical & virtual chunks. This ignores static data.

impl ChunkStore

pub fn latest_at_relevant_chunks( &self, report_mode: ChunkTrackingMode, query: &LatestAtQuery, entity_path: &EntityPath, component: ComponentIdentifier, ) -> QueryResults

Returns the most-relevant chunk(s) for the given LatestAtQuery and ComponentIdentifier.

The returned vector is guaranteed free of duplicates, by definition.

The ChunkStore always work at the Chunk level (as opposed to the row level): it is oblivious to the data therein. For that reason, and because Chunks are allowed to temporally overlap, it is possible that a query has more than one relevant chunk.

The caller should filter the returned chunks further (see Chunk::latest_at) in order to determine what exact row contains the final result.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn latest_at_relevant_chunks_for_all_components( &self, report_mode: ChunkTrackingMode, query: &LatestAtQuery, entity_path: &EntityPath, include_static: bool, ) -> QueryResults

Returns the most-relevant chunk(s) for the given LatestAtQuery.

Optionally include static data.

The ChunkStore always work at the Chunk level (as opposed to the row level): it is oblivious to the data therein. For that reason, and because Chunks are allowed to temporally overlap, it is possible that a query has more than one relevant chunk.

The returned vector is free of duplicates.

The caller should filter the returned chunks further (see Chunk::latest_at) in order to determine what exact row contains the final result.

impl ChunkStore

pub fn range_relevant_chunks( &self, report_mode: ChunkTrackingMode, query: &RangeQuery, entity_path: &EntityPath, component: ComponentIdentifier, ) -> QueryResults

Returns the most-relevant chunk(s) for the given RangeQuery and ComponentIdentifier.

The returned vector is guaranteed free of duplicates, by definition.

The criteria for returning a chunk is only that it may contain data that overlaps with the queried range.

The caller should filter the returned chunks further (see Chunk::range) in order to determine how exactly each row of data fit with the rest.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn range_relevant_chunks_for_all_components( &self, report_mode: ChunkTrackingMode, query: &RangeQuery, entity_path: &EntityPath, include_static: bool, ) -> QueryResults

Returns the most-relevant chunk(s) for the given RangeQuery.

The criteria for returning a chunk is only that it may contain data that overlaps with the queried range, or that it is static.

The returned vector is free of duplicates.

The caller should filter the returned chunks further (see Chunk::range) in order to determine how exactly each row of data fit with the rest.

impl ChunkStore

pub fn stats(&self) -> ChunkStoreStats

Returns the physical stats for this store.

I.e. this only accounts for chunks that are physically loaded in memory.

impl ChunkStore

Entity stats

pub fn entity_stats_static( &self, entity_path: &EntityPath, ) -> ChunkStoreChunkStats

Physical stats about all chunks with static data for an entity.

I.e. this only accounts for chunks that are physically loaded in memory.

pub fn entity_stats_on_timeline( &self, entity_path: &EntityPath, timeline: &TimelineName, ) -> ChunkStoreChunkStats

Physical stats about all the chunks that has data for an entity on a specific timeline.

I.e. this only accounts for chunks that are physically loaded in memory.

Does NOT include static data.

impl ChunkStore

Component path stats

pub fn num_physical_static_events_for_component( &self, entity_path: &EntityPath, component: ComponentIdentifier, ) -> u64

Returns the number of physical static events logged for an entity for a specific component.

I.e. this only accounts for chunks that are physically loaded in memory.

This ignores temporal events.

pub fn num_physical_temporal_events_for_component_on_timeline( &self, timeline: &TimelineName, entity_path: &EntityPath, component: ComponentIdentifier, ) -> u64

Returns the number of physical temporal events logged for an entity for a specific component on a given timeline.

I.e. this only accounts for chunks that are physically loaded in memory.

This ignores static events.

pub fn num_physical_temporal_events_for_component_on_all_timelines( &self, entity_path: &EntityPath, component: ComponentIdentifier, ) -> u64

Returns the number of physical temporal events logged for an entity for a specific component on all timelines.

I.e. this only accounts for chunks that are physically loaded in memory.

This ignores static events.

impl ChunkStore

pub fn new(id: StoreId, config: ChunkStoreConfig) -> ChunkStore

Instantiate a new empty ChunkStore with the given ChunkStoreConfig.

See also:

• ChunkStore::new
• ChunkStore::from_rrd_reader

pub fn new_handle(id: StoreId, config: ChunkStoreConfig) -> ChunkStoreHandle

Instantiate a new empty ChunkStore with the given ChunkStoreConfig.

Pre-wraps the result in a ChunkStoreHandle.

See also:

• ChunkStore::new

pub fn id(&self) -> StoreId

pub fn generation(&self) -> ChunkStoreGeneration

Return the current ChunkStoreGeneration. This can be used to determine whether the database has been modified since the last time it was queried.

pub fn config(&self) -> &ChunkStoreConfig

See ChunkStoreConfig for more information about configuration.

pub fn entity_tree(&self) -> &EntityTree

The hierarchical tree of all entities registered in the store.

pub fn iter_physical_chunks(&self) -> impl Iterator<Item = &Arc<Chunk>>

Iterate over all physical chunks in the store, in ascending ChunkId order.

pub fn physical_chunk(&self, physical_chunk_id: &ChunkId) -> Option<&Arc<Chunk>>

Get a physical chunk based on its ID.

pub fn use_chunk_or_report_missing(&self, id: &ChunkId) -> Option<&Arc<Chunk>>

Get a physical chunk based on its ID and track the chunk as either used or missing, to signal that it should be kept or fetched.

If the given chunk isn’t physical None is returned and the ID is reported missing.

pub fn use_transient_chunk_or_report_missing( &self, id: &ChunkId, ) -> Option<&Arc<Chunk>>

Get a physical chunk based on its ID and track the chunk as either used or missing, to signal that it should be kept or fetched.

If the given chunk isn’t physical None is returned and the ID is reported missing.

Unlike ChunkStore::use_chunk_or_report_missing, this does not signal that similar chunks should also be downloaded.

pub fn use_chunk_or_indicate(&self, id: &ChunkId) -> Option<&Arc<Chunk>>

Get a physical chunk based on its ID and track the chunk as either used or indicated, to signal that it should be kept or fetched.

If the given chunk isn’t physical None is returned and the ID is reported as possibly needed in the future.

Unlike ChunkStore::use_chunk_or_report_missing, this does make missing chunks required.

pub fn num_physical_chunks(&self) -> usize

Get the number of physical chunks in the store.

pub fn schema(&self) -> &StoreSchema

The incrementally maintained store schema.

Contains all column descriptors, per-entity component sets, timeline types, and per-column metadata.

pub fn physical_chunks(&self) -> impl Iterator<Item = &Arc<Chunk>>

All the currently loaded chunks

pub fn lookup_column_metadata( &self, entity_path: &EntityPath, component: ComponentIdentifier, ) -> Option<ColumnMetadata>

Lookup the ColumnMetadata for a specific EntityPath and re_types_core::Component.

pub fn take_tracked_chunk_ids(&self) -> QueriedChunkIdTracker

Returns and iterator over ChunkIds that were detected as used or missing since the last time since method was called.

Chunks are considered missing when they are required to compute the results of a query, but cannot be found in local memory.

Calling this method is destructive: the internal set is cleared on every call, and will grow back as new queries are run. Callers are expected to call this once per frame in order to know which chunks were missing during the previous frame.

The returned ChunkIds can live anywhere within the lineage tree, and therefore might not be usable for downstream consumers that did not track even compaction/split-off events. Use Self::find_root_chunks to find the original chunks that those IDs descended from.

pub fn tracked_chunk_ids(&self) -> QueriedChunkIdTracker

See Self::take_tracked_chunk_ids for more details.

pub fn report_used_physical_chunk_id(&self, chunk_id: ChunkId)

Signal that the chunk was used and should not be evicted by gc.

pub fn report_missing_virtual_chunk_id(&self, chunk_id: ChunkId)

Signal that a chunk is missing and should be fetched when possible.

pub fn report_transient_used_physical_chunk_id(&self, chunk_id: ChunkId)

Signal that the chunk was used and should not be evicted by gc.

Unlike ChunkStore::report_used_physical_chunk_id, this does not signal that similar chunks should also be downloaded.

pub fn report_transient_missing_virtual_chunk_id(&self, chunk_id: ChunkId)

Signal that a chunk is missing and should be fetched when possible.

Unlike ChunkStore::report_missing_virtual_chunk_id, this does not signal that similar chunks should also be downloaded.

pub fn indicate_virtual_chunk_id(&self, chunk_id: ChunkId)

Signal that a chunk should be fetched when possible.

Unlike ChunkStore::report_missing_virtual_chunk_id, this does not make the missing chunk required.

pub fn num_missing_chunk_ids(&self) -> usize

How many missing chunk IDs are currently registered?

See also ChunkStore::take_tracked_chunk_ids.

impl ChunkStore

pub fn from_rrd_reader( store_config: &ChunkStoreConfig, reader: &mut dyn Read, ) -> Result<BTreeMap<StoreId, ChunkStore>, Error>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

The stores will be prefilled with the data from the given RRD reader.

See also:

• ChunkStore::new

pub async fn from_rrd_reader_async( store_config: &ChunkStoreConfig, reader: &mut (dyn AsyncRead + Unpin + Send), ) -> Result<BTreeMap<StoreId, ChunkStore>, Error>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

The stores will be prefilled with the data from the given RRD reader.

See also:

• ChunkStore::new

pub fn from_log_msgs( store_config: &ChunkStoreConfig, log_msgs: impl IntoIterator<Item = LogMsg>, ) -> Result<BTreeMap<StoreId, ChunkStore>, Error>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

The stores will be prefilled with the data in the given log_msgs.

See also:

• ChunkStore::new

pub fn handle_from_rrd_reader( store_config: &ChunkStoreConfig, reader: impl Read, ) -> Result<BTreeMap<StoreId, ChunkStoreHandle>, Error>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

Wraps the results in ChunkStoreHandles.

The stores will be prefilled with the data from the given RRD reader.

See also:

• ChunkStore::new_handle

pub async fn handle_from_rrd_reader_async<R>( store_config: &ChunkStoreConfig, reader: R, ) -> Result<BTreeMap<StoreId, ChunkStoreHandle>, Error>

where R: AsyncRead + Unpin + Send,

Instantiate new ChunkStoreHandles with the given ChunkStoreConfig.

The stores will be prefilled with the data from the given RRD reader.

See also:

• ChunkStore::new_handle

impl ChunkStore

pub fn register_subscriber( subscriber: Box<dyn ChunkStoreSubscriber>, ) -> ChunkStoreSubscriberHandle

Registers a ChunkStoreSubscriber so it gets automatically notified when data gets added and/or removed to/from a ChunkStore.

Refer to ChunkStoreEvent’s documentation for more information about these events.

Scope

Registered ChunkStoreSubscribers are global scope: they get notified of all events from all existing ChunkStores, including ChunkStores created after the subscriber was registered.

Use ChunkStoreEvent::store_id to identify the source of an event.

Late registration

Subscribers must be registered before a store gets created to guarantee that no events were missed.

ChunkStoreEvent::event_id can be used to identify missing events.

Ordering

The order in which registered subscribers are notified is undefined and will likely become concurrent in the future.

If you need a specific order across multiple subscribers, embed them into an orchestrating subscriber.

pub fn with_subscriber<V, T, F>( _: ChunkStoreSubscriberHandle, f: F, ) -> Option<T>

where V: ChunkStoreSubscriber, F: FnMut(&V) -> T,

Passes a reference to the downcasted subscriber to the given FnMut callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_once<V, T, F>( _: ChunkStoreSubscriberHandle, f: F, ) -> Option<T>

where V: ChunkStoreSubscriber, F: FnOnce(&V) -> T,

Passes a reference to the downcasted subscriber to the given FnOnce callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_mut<V, T, F>( _: ChunkStoreSubscriberHandle, f: F, ) -> Option<T>

where V: ChunkStoreSubscriber, F: FnMut(&mut V) -> T,

Passes a mutable reference to the downcasted subscriber to the given callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn register_per_store_subscriber<S>() -> ChunkStoreSubscriberHandle

where S: PerStoreChunkSubscriber + Default + 'static,

Registers a PerStoreChunkSubscriber type so it gets automatically notified when data gets added and/or removed to/from a ChunkStore.

pub fn drop_per_store_subscribers(store_id: &StoreId)

Notifies all PerStoreChunkSubscribers that a store was dropped.

pub fn with_per_store_subscriber<S, T, F>( _: ChunkStoreSubscriberHandle, store_id: &StoreId, f: F, ) -> Option<T>

where S: PerStoreChunkSubscriber + 'static, F: FnMut(&S) -> T,

Passes a reference to the downcasted per-store subscriber to the given FnMut callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_per_store_subscriber_once<S, T, F>( _: ChunkStoreSubscriberHandle, store_id: &StoreId, f: F, ) -> Option<T>

where S: PerStoreChunkSubscriber + 'static, F: FnOnce(&S) -> T,

Passes a reference to the downcasted per-store subscriber to the given FnOnce callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_per_store_subscriber_mut<S, T, F>( _: ChunkStoreSubscriberHandle, store_id: &StoreId, f: F, ) -> Option<T>

where S: PerStoreChunkSubscriber + 'static, F: FnMut(&mut S) -> T,

Passes a mutable reference to the downcasted per-store subscriber to the given callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn capture_all_subscribers_mem_usage_tree() -> MemUsageTree

Captures the memory usage of all registered subscribers.

Names are disambiguated with a #idx suffix when multiple subscribers share the same name.

impl ChunkStore

pub fn insert_rrd_manifest( &mut self, rrd_manifest: Arc<RrdManifest>, ) -> Vec<ChunkStoreEvent>

This insert a batch of virtual chunks into the store, according to the given RrdManifest.

All queries will return partial results until the missing physical data gets loaded in.

pub fn insert_chunk( &mut self, chunk: &Arc<Chunk>, ) -> Result<Vec<ChunkStoreEvent>, ChunkStoreError>

Inserts a Chunk in the store.

Iff the store was modified, all registered subscribers will be notified and the resulting ChunkStoreEvent will be returned, or None otherwise.

• Trying to insert an unsorted chunk (Chunk::is_sorted) will fail with an error.
• Inserting a duplicated ChunkId will result in a no-op.
• Inserting an empty Chunk will result in a no-op.

pub fn drop_entity_path( &mut self, entity_path: &EntityPath, ) -> Vec<ChunkStoreEvent>

Unconditionally drops all the data for a given entity_path.

Returns the list of Chunks that were dropped from the store in the form of ChunkStoreEvents.

This is not recursive. The store is unaware of the entity hierarchy.

Trait Implementations

impl Clone for ChunkStore

fn clone(&self) -> ChunkStore

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for ChunkStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Drop for ChunkStore

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl MemUsageTreeCapture for ChunkStore

fn capture_mem_usage_tree(&self) -> MemUsageTree

impl SizeBytes for ChunkStore

fn heap_size_bytes(&self) -> u64

Returns how many bytes self uses on the heap.

const IS_POD: bool = false

fn total_size_bytes(&self) -> u64

Returns the total size of self in bytes, accounting for both stack and heap space.

fn stack_size_bytes(&self) -> u64

Returns the total size of self on the stack, in bytes.