Skip to main content

DeduplicationDataInterface

xet_data::deduplication

Trait DeduplicationDataInterface 

Source 
pub trait DeduplicationDataInterface:
    Send
    + Sync
    + 'static {
    type ErrorType;

    // Required methods
    fn chunk_hash_dedup_query<'life0, 'life1, 'async_trait>(
        &'life0 self,
        query_hashes: &'life1 [MerkleHash],
    ) -> Pin<Box<dyn Future<Output = Result<Option<(usize, FileDataSequenceEntry, bool)>, Self::ErrorType>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait;
    fn register_global_dedup_query<'life0, 'async_trait>(
        &'life0 mut self,
        _chunk_hash: MerkleHash,
    ) -> Pin<Box<dyn Future<Output = Result<(), Self::ErrorType>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn complete_global_dedup_queries<'life0, 'async_trait>(
        &'life0 mut self,
    ) -> Pin<Box<dyn Future<Output = Result<bool, Self::ErrorType>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn register_new_xorb<'life0, 'async_trait>(
        &'life0 mut self,
        xorb: RawXorbData,
    ) -> Pin<Box<dyn Future<Output = Result<(), Self::ErrorType>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn register_xorb_dependencies<'life0, 'life1, 'async_trait>(
        &'life0 mut self,
        dependencies: &'life1 [FileXorbDependency],
    ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait;
}
Expand description

The interface needed for the deduplication routines to run. To use the deduplication code, define a struct that implements these methods. This struct must be given by value to the FileDeduper struct on creation.

The two primary methods are chunk_hash_dedup_query, which determines whether and how a chunk can be deduped,
and register_new_xorb, which is called intermittently when a new block of data is available for upload.

The global dedup query functions are optional but needed if global dedup is to be enabled.

Required Associated Types§

Source

type ErrorType

The error type used for the interface

Required Methods§

Source

fn chunk_hash_dedup_query<'life0, 'life1, 'async_trait>( &'life0 self, query_hashes: &'life1 [MerkleHash], ) -> Pin<Box<dyn Future<Output = Result<Option<(usize, FileDataSequenceEntry, bool)>, Self::ErrorType>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Query for possible shards that

Source

fn register_global_dedup_query<'life0, 'async_trait>( &'life0 mut self, _chunk_hash: MerkleHash, ) -> Pin<Box<dyn Future<Output = Result<(), Self::ErrorType>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Registers a new query for more information about the global deduplication. This is expected to run in the background. Simply return Ok(()) to disable global dedup queries.

Source

fn complete_global_dedup_queries<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<bool, Self::ErrorType>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Waits for all the current queries to complete, then returns true if there is new deduplication information available.

Source

fn register_new_xorb<'life0, 'async_trait>( &'life0 mut self, xorb: RawXorbData, ) -> Pin<Box<dyn Future<Output = Result<(), Self::ErrorType>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Registers a Xorb of new data that has no deduplication references.

Source

fn register_xorb_dependencies<'life0, 'life1, 'async_trait>( &'life0 mut self, dependencies: &'life1 [FileXorbDependency], ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Register a set of xorb dependencies; this is called periodically during the dedup process with a list of (xorb hash, n_bytes). As the final bit may get returned as a partial xorb without a hash yet, it is not gauranteed that the sum of the n_bytes across all the dependencies will equal the size of the file.

Implementors§