Crate vmap

source ·
Expand description

A cross-platform library for fast and safe memory-mapped IO and boundary-free ring buffer.

This library defines a convenient API for reading and writing to files using the hosts virtual memory system, as well as allocating memory and creating circular memory regions. The design of the API strives to both minimize the frequency of mapping system calls while still retaining safe access. Critically, it never attempts the own the File object used for mapping. That is, it never clones it or in any way retains it. While this has some implications for the API (i.e. .flush()), it cannot cause bugs outside of this library through File’s leaky abstraction when cloned and then closed.

The Map and MapMut types are primary means for allocating virtual memory regions, both for a file and anonymously. Generally, the Map::with_options() and MapMut::with_options() are used to specify the mapping requirements. See Options for more information.

The MapMut type maintains interior mutability for the mapped memory, while the Map is read-only. However, it is possible to convert between these types (.into_map_mut() and .into_map()) assuming the proper Options are specified.

Additionally, a variety of buffer implementations are provided in the vmap::io module. The Ring and InfiniteRing use cross-platform optimzed circular memory mapping to remove the typical boundary problem with most circular buffers. This ensures all ranges of the underlying byte buffer can be viewed as a single byte slice, event when the value wraps back around to the beginning of the buffer. The BufReader and BufWriter implement buffered I/O using a Ring as a backing layer.

§Examples

use vmap::Map;
use std::path::PathBuf;
use std::str::from_utf8;

let path: PathBuf = /* path to file */

// Map the first 4 bytes
let (map, file) = Map::with_options().len(4).open(&path)?;
assert_eq!(Ok("this"), from_utf8(&map[..]));

// Reuse the file to map a different region
let map = Map::with_options().offset(10).len(4).map(&file)?;
assert_eq!(Ok("test"), from_utf8(&map[..]));

If opened properly, the Map can be moved into a MapMut and modifications to the underlying file can be performed:

use vmap::{Map, Flush};
use std::path::PathBuf;
use std::str::from_utf8;

let path: PathBuf = /* path to file */

// Open with write permissions so the Map can be converted into a MapMut
let (map, file) = Map::with_options().write().len(14).open(&path)?;
assert_eq!(Ok("this is a test"), from_utf8(&map[..]));

// Move the Map into a MapMut
// ... we could have started with MapMut::with_options()
let mut map = map.into_map_mut()?;
map[..4].clone_from_slice(b"that");

// Flush the changes to disk synchronously
map.flush(&file, Flush::Sync)?;

// Move the MapMut back into a Map
let map = map.into_map()?;
assert_eq!(Ok("that is a test"), from_utf8(&map[..]));

This library contains a Ring that constructs a circular memory allocation where values can wrap from around from the end of the buffer back to the beginning with sequential memory addresses. The InfiniteRing is similar, however it allows writes to overwrite reads.

use vmap::io::{Ring, SeqWrite};
use std::io::{BufRead, Read, Write};

let mut buf = Ring::new(4000).unwrap();
let mut i = 1;

// Fill up the buffer with lines.
while buf.write_len() > 20 {
    write!(&mut buf, "this is test line {}\n", i)?;
    i += 1;
}

// No more space is available.
assert!(write!(&mut buf, "this is test line {}\n", i).is_err());

let mut line = String::new();

// Read the first line written.
let len = buf.read_line(&mut line)?;
assert_eq!(line, "this is test line 1\n");

line.clear();

// Read the second line written.
let len = buf.read_line(&mut line)?;
assert_eq!(line, "this is test line 2\n");

// Now there is enough space to write more.
write!(&mut buf, "this is test line {}\n", i)?;

Modules§

  • Read/Write types for buffering.
  • Low-level cross-platform virtual memory functions

Structs§

  • A list specifying general categories of map errors.
  • Allocation of one or more read-only sequential pages.
  • Allocation of one or more read-write sequential pages.
  • Options and flags which can be used to configure how a map is allocated.
  • Type for calculation system page or allocation size information.

Enums§

  • Hint for the access pattern of the underlying mapping.
  • Byte extent type used for length and resize options.
  • Desired behavior when flushing write changes.
  • A list specifying general categories of input mapping errors.
  • A list specifying general categories of erroneous operations.
  • Protection level for a page.

Traits§

  • General trait for working with any memory-safe representation of a contiguous region of arbitrary memory.
  • General trait for working with any memory-safe representation of a contiguous region of arbitrary mutable memory.

Functions§

  • Gets a cached version of the system allocation granularity size.
  • Gets a cached version of the system page size.

Type Aliases§

  • A specialiazed Result type for conversion operations.
  • A specialized Result type for map operations.