hyperfile 0.10.4

A file-like interface abstraction layer provides random read and write capabilities for S3
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

//! Blocks whose bytes live outside the container until they are uploaded.
//!
//! A caller that already keeps a durable copy of what it writes — a block device
//! that stages on local disk, say — does not need the container to hold a second
//! copy in memory for the whole life of a checkpoint. It can hand over a
//! *reference* instead: [`HyperFile::write_ref`](crate::file::file::HyperFile::write_ref)
//! records that a range of the file is to be read from a [`BlockSource`] when the
//! checkpoint is built, and the bytes are only fetched as each part of the segment
//! is uploaded.
//!
//! What goes to storage is the same segment, byte for byte, that writing the data
//! would have produced: a reference changes where the bytes come from at upload
//! time, never what is stored or how it is laid out. A container written this way
//! is read by any version that reads the format.
//!
//! The contract is on the source: from `write_ref` until the flush that publishes
//! the checkpoint returns, reading a referenced range must yield the bytes the
//! caller meant at the time of `write_ref`, even if the caller has since written
//! the same range again (that later write is the caller's to keep apart — see the
//! `write_ref` documentation).

use std::io::Result;

/// Where referenced blocks are read from.
///
/// `off` is an offset in the **file's** address space; the source maps it to
/// wherever it keeps the bytes. Reads are always whole data blocks.
///
/// Called from a blocking thread while a segment is being uploaded, so it may do
/// ordinary blocking I/O.
pub trait BlockSource: Send + Sync {
    fn read_at(&self, off: u64, buf: &mut [u8]) -> Result<()>;
}