Crate vmap

A cross-platform library for fast and safe memory-mapped IO

This library defines a convenient API for reading and writing to files using the hosts virtual memory system. The design of the API strives to both minimize the frequency of mapping system calls while still retaining safe access. Critically, it never attempts the own the File object used for mapping. That is, it never clones it or in any way retains it. While this has some implications for the API (i.e. .flush()), it cannot cause bugs outside of this library through File's leaky abstraction when cloned and then closed.

The Map and MapMut types are primary means for allocating virtual memory regions, both for a file and anonymously. Generally, the Map::with_options() and MapMut::with_options() are used to specify the mapping requirements. See Options for more information.

The MapMut type maintains interior mutability for the mapped memory, while the Map is read-only. However, it is possible to convert between these types (.into_map_mut() and .into_map()) assuming the proper Options are specified.

Additionally, a variety of buffer implementations are provided in the vmap::io module. The Ring and InfiniteRing use circular memory address allocations using cross-platform optimizations to minimize excess resources where possible. The BufReader and BufWriter implement buffered I/O using a Ring as a backing layer.

Examples

use vmap::Map;
use std::path::PathBuf;
use std::str::from_utf8;

let path: PathBuf = /* path to file */

// Map the first 4 bytes
let (map, file) = Map::with_options().len(4).open(&path)?;
assert_eq!(Ok("this"), from_utf8(&map[..]));

// Reuse the file to map a different region
let map = Map::with_options().offset(10).len(4).map(&file)?;
assert_eq!(Ok("test"), from_utf8(&map[..]));

If opened properly, the Map can be moved into a MapMut and modifications to the underlying file can be performed:

use vmap::{Map, Flush};
use std::path::PathBuf;
use std::str::from_utf8;

let path: PathBuf = /* path to file */

// Open with write permissions so the Map can be converted into a MapMut
let (map, file) = Map::with_options().write().len(14).open(&path)?;
assert_eq!(Ok("this is a test"), from_utf8(&map[..]));

// Move the Map into a MapMut
// ... we could have started with MapMut::with_options()
let mut map = map.into_map_mut()?;
map[..4].clone_from_slice(b"that");

// Flush the changes to disk synchronously
map.flush(&file, Flush::Sync)?;

// Move the MapMut back into a Map
let map = map.into_map()?;
assert_eq!(Ok("that is a test"), from_utf8(&map[..]));

This library contains a Ring that constructs a circular memory allocation where values can wrap from around from the end of the buffer back to the beginning with sequential memory addresses. The InfiniteRing is similar, however it allows writes to overwrite reads.

use vmap::io::{Ring, SeqWrite};
use std::io::{BufRead, Read, Write};

let mut buf = Ring::new(4000).unwrap();
let mut i = 1;

// Fill up the buffer with lines.
while buf.write_len() > 20 {
    write!(&mut buf, "this is test line {}\n", i)?;
    i += 1;
}

// No more space is available.
assert!(write!(&mut buf, "this is test line {}\n", i).is_err());

let mut line = String::new();

// Read the first line written.
let len = buf.read_line(&mut line)?;
assert_eq!(line, "this is test line 1\n");

line.clear();

// Read the second line written.
let len = buf.read_line(&mut line)?;
assert_eq!(line, "this is test line 2\n");

// Now there is enough space to write more.
write!(&mut buf, "this is test line {}\n", i)?;

Modules

io	Read/Write types for buffering.
os	Low-level cross-platform virtual memory functions

Structs

Error	A list specifying general categories of map errors.
Map	Allocation of one or more read-only sequential pages.
MapMut	Allocation of one or more read-write sequential pages.
Options	Options and flags which can be used to configure how a map is allocated.
Size	Type for calculation system page or allocation size information.

Enums

AdviseAccess	Hint for the access pattern of the underlying mapping.
AdviseUsage	Hint for the immediacy of accessing the underlying mapping.
Extent	Byte extent type used for length and resize options.
Flush	Desired behavior when flushing write changes.
Input	A list specifying general categories of input mapping errors.
Operation	A list specifying general categories of erroneous operations.
Protect	Protection level for a page.

Traits

Span	General trait for working with any memory-safe representation of a contiguous region of arbitrary memory.
SpanMut	General trait for working with any memory-safe representation of a contiguous region of arbitrary mutable memory.

Functions

allocation_size	Gets a cached version of the system allocation granularity size.
page_size	Gets a cached version of the system page size.

Type Definitions

AllocSize	Deprecated Type for calculation system page or allocation size information.
ConvertResult	A specialiazed Result type for conversion operations.
Result	A specialized Result type for map operations.