Skip to main content

Store

Struct Store 

Source
pub struct Store(/* private fields */);
Available on Linux or macOS or Windows only.
Expand description

A store: one container file in a directory of its own, on one device.

Every method of store_io_engine::Store is available through Deref: provisioning and looking up regions and slots, reserve, space, report, poison.

use store_io::{Store, StoreOptions};

let store = Store::create("data/orders", StoreOptions::default())?;
println!("{}", store.report());
drop(store);
let store = Store::open("data/orders")?;
let wal = store.append_region("wal");

Implementations§

Source§

impl Store

Source

pub fn create(path: impl AsRef<Path>, opts: StoreOptions) -> Result<Self, Error>

Creates a new store in path (the directory is created if needed). The device is probed first: unsafe or unverified devices are refused unless opts.trust attests or overrides (and then every receipt is labelled accordingly).

§Errors

AlreadyExists, UnsafeDevice, Unverified, NoSpace, Io.

Source

pub fn open(path: impl AsRef<Path>) -> Result<Self, Error>

Opens an existing store for writing, with default options. Takes the ownership lock, waits out a dead owner’s writes, and verifies every region against its header. Append regions must be positioned with resume_at before the first append.

§Errors

NotFound, Locked, UnsafeDevice, Unverified, Corruption, DurabilityUnknown (a dead owner’s writes could not be made durable), Io; Unsupported(FormatVersion) for a newer format and for a store of container format 1 (before 0.5), which opens read-only (Self::open_readonly) and moves to format 2 by copy_store (store-io copy).

Source

pub fn open_with( path: impl AsRef<Path>, opts: StoreOptions, ) -> Result<Self, Error>

Opens an existing store for writing with explicit options.

§Errors

As Self::open.

Source

pub fn open_readonly(path: impl AsRef<Path>) -> Result<Self, Error>

Opens an existing store for reading and recovery only: no write rights, and nothing on disk changes.

§Errors

NotFound, Corruption, Io.

Source

pub fn open_readonly_with( path: impl AsRef<Path>, opts: StoreOptions, ) -> Result<Self, Error>

Opens an existing store for reading and recovery only, with explicit options (for example linux_tier: LinuxTier::Sync).

§Errors

As Self::open_readonly.

Source

pub fn engine(&self) -> &Store<NativePlatform>

The engine store underneath.

Methods from Deref<Target = Store<NativePlatform>>§

Source

pub fn export( &self, sink: &mut dyn ExportSink, ) -> Result<ExportReport, Box<ExportError>>

Exports every region to sink, in the store’s order: see the module docs and Self::export_with.

§Errors

As Self::export_with.

Source

pub fn export_with( &self, sink: &mut dyn ExportSink, opts: ExportOptions, ) -> Result<ExportReport, Box<ExportError>>

Exports the regions opts names (all by default) to sink: its begin with the header, then for each region its region, its bytes through data and its end_region, then its finish.

§Errors

An ExportError (boxed: it holds the report) with how far it got. Before the sink is called: NotWritten(InvalidOptions) for a region named twice, a name no region has, a depth out of range, or on a writable open a region that only a read-only open exports (one of a stale page, possibly superseded, or at risk: open the store read-only to export it); NotWritten(NotReady) for an unavailable region or an entry that does not decode, NotWritten(Frozen) for a stale or lost table page (both unless ExportOptions::accept_stale_pages); NotWritten(TooLarge) for a slot object a format-2 slot cannot hold; NotWritten(StaleGeneration) for a region that changed while the plan read it; NotWritten(PoolExhausted) with too few buffers free for the stream; Io if its queue could not be made. At a region’s turn: NotWritten(StaleGeneration) (recycled) or NotWritten(NotReady) (released) since the plan or while it streams, NotWritten(TooLarge) for a slot object committed since that a format-2 slot cannot hold; a read’s errors as a ReadStream’s, a media error (Corruption(MediaError), the range named) among them; NotWritten(TooManyInFlight) if the sink still holds every lent chunk after settle. Any error of the sink (in sink).

Source

pub fn append_region(&self, name: &str) -> Result<AppendRegion<P>, Error>

Looks up an existing append region.

§Errors

Error::NotFound if no ready append region has that name.

Source

pub fn provision_append_region( &self, name: &str, size: u64, ) -> Result<AppendRegion<P>, Error>

Provisions a new append region of at least size bytes.

§Errors

Error::AlreadyExists, Error::NoSpace, or a platform error.

Source

pub fn provision_append_region_with( &self, name: &str, opts: RegionOptions, ) -> Result<AppendRegion<P>, Error>

Provisions a new append region as opts say (its size, and in container format 2 its cap).

§Errors

As Self::provision_append_region; NotWritten(InvalidOptions) for options RegionOptions::validate refuses; Unsupported for a cap in container format 1.

Source

pub fn provision_append_region_default( &self, name: &str, ) -> Result<AppendRegion<P>, Error>

Provisions a new append region with the store’s StoreOptions::region_defaults.

let opts = StoreOptions {
    region_defaults: Some(RegionOptions::default().size(1 << 20)),
    ..StoreOptions::default()
};
let store = Store::create(SimPlatform::new(SimConfig::volatile(1)), Path::new("/db"), opts)?;
let log = store.provision_append_region_default("log")?;
assert_eq!(log.len(), 1 << 20);
§Errors

As Self::provision_append_region_with; NotWritten(InvalidOptions) when the store has no defaults (before any other refusal).

Source

pub fn provision_page_region_default( &self, name: &str, ) -> Result<PageRegion<P>, Error>

Provisions a new page region with the store’s StoreOptions::region_defaults.

§Errors

As Self::provision_append_region_default.

Source

pub fn page_region(&self, name: &str) -> Result<PageRegion<P>, Error>

Looks up an existing page region.

§Errors

Error::NotFound if no ready page region has that name.

Source

pub fn provision_page_region( &self, name: &str, size: u64, ) -> Result<PageRegion<P>, Error>

Provisions a new page region of at least size bytes.

§Errors

Error::AlreadyExists, Error::NoSpace, or a platform error.

Source

pub fn provision_page_region_with( &self, name: &str, opts: RegionOptions, ) -> Result<PageRegion<P>, Error>

Provisions a new page region as opts say (its size, and in container format 2 its cap).

§Errors

As Self::provision_append_region_with.

Source

pub fn slot(&self, name: &str) -> Result<Slot<P>, Error>

Looks up an existing small-object slot.

§Errors

Error::NotFound if no slot has that name.

Source

pub fn provision_slot(&self, name: &str) -> Result<Slot<P>, Error>

Provisions a new small-object slot. Its capacity is the block size minus 176 bytes of headers (Slot::capacity).

§Errors

Error::AlreadyExists, Error::NoSpace, or a platform error.

Source

pub fn repair_unavailable(&self, id: RegionId) -> Result<(), Error>

Releases the table entry of an unavailable region, one that Self::unavailable_regions lists (or, named by its id page << 32 | seq, an entry of a kind this version does not know, report().metadata.invalid_entries: given up deliberately). The entry’s page is written without it (durable on return); the region’s bytes are not touched, and its space becomes free where no other region’s footprint lies. Its name is free again; the repair is recorded (Self::repairs). Container format 2.

§Errors

NotFound(Region) if id names no unavailable entry (a usable region is released with release); NotWritten(Overlap), naming a region whose open-time verdict rests on this entry (spec §9 step 4: a live region it supersedes, which may read another’s bytes, or an unavailable one it demoted by an overlap or a name): the entry stays while that region does (repair a demoted one first; a superseded one is read-only and cannot be released: the way out is a copy of the store that leaves unavailable entries out, copy_store’s accept_stale_pages, v0.5 E6 C4); NotWritten(Frozen) also for a page holding an entry past its own write that would be kept (repair that one first, when it can be: one that supersedes a live region cannot, and two on one page block each other; the way out is then the copy); NotWritten(Frozen) while allocation is frozen (a table page is stale or lost: free space cannot be trusted); NotWritten(ReadOnly); Unsupported(FormatVersion) for format 1; Poisoned; DurabilityUnknown if the page write failed after submission (the store is then poisoned; the entry may or may not be gone at the next open).

Source

pub fn accept_page_rollback(&self, pages: &[u32]) -> Result<(), Error>

Accepts the rollback of stale table pages (spec §12): for each page of pages that is Unconfirmed (report().metadata.stale_pages), its winner is rewritten byte for byte into its failed slot (even an unreadable one: writing usually remaps the sector) at the next meta_seq, durably, and read back; once the new copy reads back as the winner the page counts as Confirmed: whatever its lost commits held is given up (space they used may be handed out again) and the extra extents counted allocated for it are free. Every page is checked before any is written (an error from a check accepts none). Each accepted page is recorded (Self::repairs); a Confirmed page is left as it is.

Once no page is stale or lost, allocation is no longer frozen (Self::frozen) and the regions become writable (not one possibly superseded, or with a header feature this version may not write). Not before: a region of an accepted page may be one a lost commit released, its space then given to a region of another stale page’s lost commit, whose bytes it would overwrite. The same holds across a reopen: an accepted page’s two slots then hold the same payload (as after a heal), and while any page is stale or lost the open keeps such a page’s regions read-only (also once its older copy is damaged: any Confirmed page with a damaged older copy counts). A store with a Lost page stays frozen, the regions of the pages accepted read-only: it is copied (copy_store). An accepted page’s next_seq counts higher (raised by the open that found it stale and by each open while its trace lasts, written by its next commit): the region ids its lost commit gave out are not given out again (unless more than 4,096 provisionings failed on that page between its last two commits). Container format 2.

§Errors

NotWritten(NotReady) for a Lost page, its slots named (no winner to keep); NotWritten(OutOfBounds) for a page the table does not have; NotWritten(Frozen) for a page holding an entry whose footprint_seq is past the page’s own write, the entry named (the rewrite would change how the next open counts it, and no repair runs while allocation is frozen: such a store is copied). Corruption(MediaError) if the page does not read back with both slots Valid and the same, and Io if it does not read: the write was durable, so the store is poisoned (what the next open finds is the page’s state: Confirmed if the new copy reads, else stale, to be accepted again). NotWritten(ReadOnly); Unsupported(FormatVersion) for format 1; Poisoned; DurabilityUnknown if the write failed after submission (the store poisoned; at the next open the page is still stale, or Confirmed if the new copy reached the media).

Source

pub fn extent_of(&self, data_size: u64) -> Option<u64>

Bytes a region of data_size data bytes occupies in the container: the data size rounded up to the block, plus two header blocks.

Source

pub fn set_cap(&self, tag: u32, cap: Option<u64>)

Sets (or with None removes) the hard cap of tag: the most bytes its regions and reservations may hold together. Caps live in memory; set them again after every open.

Source

pub fn reserve(&self, bytes: u64, tag: u32) -> Result<Reservation<P>, Error>

Reserves bytes for later provisioning under tag: checks the tag’s cap and grows the container so that provisioning from the reservation cannot run out of space.

§Errors

NoSpace { tag } if the cap or the file system is exceeded (nothing changes and nothing is poisoned); NotWritten(ReadOnly); Poisoned; Io for other allocation failures.

Source

pub fn space(&self) -> SpaceReport

Where space goes: the container, regions, released extents, reservations, and usage per tag.

Source

pub fn oldest_outstanding(&self) -> Option<Outstanding>

The oldest operation still in flight on the store’s queues (shared, parallel and flush) and its submitters, or None when they are all idle. O(queues + submitters), no I/O; reads the platform’s coarse clock once. A submitter’s operation counts until its completion is processed by reap or wait, a read stream’s until next() collects it.

The age counts from the first call that saw the operation, so a watchdog calling this every second learns of a hung device within a second of its deadline.

Source

pub fn poisoned(&self) -> Option<FirstCause>

The failure that poisoned the store, if one has: every write and barrier fails with Poisoned from then on. A cheap check, for a health probe or a monitor.

Source

pub fn observe(&self) -> Snapshot

Event counters and sampled per-stage latencies since this handle was opened. With feature observe off (the default) nothing is recorded and the snapshot is all zeros.

Source

pub fn report(&self) -> &StoreReport

What the store learned about its device and what it decided, as the open found it (never changed after: what repairs change since is in Self::unavailable_regions, Self::frozen and Self::repairs).

Source

pub fn commit_hold(&self) -> CommitHold

Keeps this store’s device group’s commits active until the hold drops (PLANNING R5.4), so background work on the device (scans, copies, background Submitters, of this store or another on the device) stays paced at one piece in flight per call. For a layer that commits rarely but needs every commit fast: it holds one while its transactions are open and drops it when idle. The cost: while a hold lives, background work runs at the paced rate.

Source

pub fn qos(&self) -> QosReport

This store’s device pacing now: whether pacing is on, whether the device’s commits are active, the budget, the background pieces in flight in the device group.

Source

pub fn io_path(&self) -> IoPath

The kernel interface the store’s I/O uses now, and why: as StoreReport::io, with every queue made since counted (a ring the kernel refused for a Submitter or a parallel lane, and what served instead).

Source

pub fn volume(&self) -> VolumeId

The volume identity.

Source

pub fn poison(&self)

Poisons the device domain (for a caller’s watchdog that decided the device is hung). Every later write and barrier fails with Error::Poisoned; reads keep working.

Source

pub fn domain_stats(&self) -> DomainStats

Flush-sharing counters of the device domain.

Source

pub fn submit_pool_threads(&self) -> Option<usize>

Worker threads in the submit pool that runs this store’s page batches and large reads concurrently, once this store’s first such call has started its parallel queues (the pool may have been started earlier by another store on the same platform, and stays None if they could not be started); None before that, or on a backend that needs no pool (IOCP, the simulator).

Source

pub fn regions(&self) -> Vec<(String, RegionKind)>

Names and kinds of the ready regions.

Source

pub fn unavailable_regions(&self) -> Vec<UnavailableRegion>

Regions the region table names that open could not use, with the reason (D-090), less any repaired since. Empty for a healthy store. Their names stay reserved.

Source

pub fn frozen(&self) -> bool

Whether allocation is frozen now (format 2: a table page is stale or lost; spec §9 step 2): no provisioning, no growth beyond capacity, no release, no repair. report().metadata.frozen is what the open found.

Source

pub fn repairs(&self) -> Vec<Repair>

What this handle’s repairs and accepts did, in order (Self::repair_unavailable, Self::accept_page_rollback): the store’s log of them, for the caller to record. The library prints nothing.

Source

pub fn prefetch( &self, ranges: &[StreamRange<'_, P>], opts: StreamOptions, ) -> Result<ReadStream<P>, Error>

The bytes of several ranges, of this store’s regions, as one ReadStream on one queue: each range in the order given, each in offset order (Chunk::range says which; overlapping ranges are read twice; no range, an empty stream). An append region’s range was clamped to its durable prefix when it was made. Each region’s gate is checked again here and as the stream reads it: a retire of a region ends the stream before any more of that region’s reads are delivered. An error tied to no read (TooManyInFlight, a failed wait, Interrupted) names the range being read ahead.

§Errors

NotWritten: ForeignPosition (a range of another store, or of another open of this one), StaleGeneration or NotReady (a region recycled or released), PoolExhausted, InvalidOptions; Io if its queue could not be made.

Source

pub fn submitter(&self, cfg: SubmitterConfig) -> Result<Submitter<P>, Error>

A Submitter for this thread, with its own queue of cfg.depth operations (1 to MAX_DEPTH).

§Errors

Io if the platform could not create the queue.

Trait Implementations§

Source§

impl Clone for Store

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for Store

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Deref for Store

Source§

type Target = Store<PosixPlatform>

The resulting type after dereferencing.
Source§

fn deref(&self) -> &Self::Target

Dereferences the value.
Source§

impl From<Store<PosixPlatform>> for Store

Source§

fn from(s: Store<NativePlatform>) -> Self

Converts to this type from the input type.
Source§

impl From<Store> for Store<NativePlatform>

Source§

fn from(s: Store) -> Self

Converts to this type from the input type.

Auto Trait Implementations§

§

impl !RefUnwindSafe for Store

§

impl !UnwindSafe for Store

§

impl Forget for Store

§

impl Freeze for Store

§

impl Move for Store

§

impl Send for Store

§

impl Sync for Store

§

impl Unpin for Store

§

impl UnsafeUnpin for Store

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<P, T> Receiver for P
where P: Deref<Target = T> + ?Sized, T: ?Sized,

Source§

type Target = T

🔬This is a nightly-only experimental API. (arbitrary_self_types)
The target type on which the method may be called.
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.