Skip to main content

Crate frozen_core

Crate frozen_core 

Source
Expand description

License Latest Version Tests Pull Requests GitHub Issues or Pull Requests

§FrozenCore

Custom implementations and core utilities for frozen codebases.

§Index

§Usage

Add following to your Cargo.toml,

[dependencies]
frozen-core = { version = "0.0.11", default-features = true }

[!TIP] All the features are enabled by default. To disable this, set default-features = false

§FrozenFile

FrozenFile is a custom implementation of std::fs::File.

To use the ffile module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["ffile"] }

FrozenFile is currently available on the following platforms,

PlatformSupport
aarch64-unknown-linux-gnu✅
x86_64-unknown-linux-gnu✅
aarch64-pc-windows-msvc❌
x86_64-pc-windows-msvc❌
aarch64-apple-darwin✅
x86_64-apple-darwin✅

See following example for more details.

§FrozenMMap

FrozenMMap is a custom implementation of mmap.

To use the fmmap module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["fmmap"] }

FrozenMMap is currently available on the following platforms,

PlatformSupport
aarch64-unknown-linux-gnu✅
x86_64-unknown-linux-gnu✅
aarch64-pc-windows-msvc❌
x86_64-pc-windows-msvc❌
aarch64-apple-darwin✅
x86_64-apple-darwin✅

Read the example below for usage details,

use frozen_core::fmmap::{FMCfg, FrozenMMap};

const MID: u8 = 0;

#[repr(C)]
#[derive(Debug)]
struct Meta(u64);

unsafe impl Send for Meta {}
unsafe impl Sync for Meta {}

fn main() {
    let dir = std::path::PathBuf::from("./target/examples");
    let path = dir.join("fm_example.bin");

    let _ = std::fs::remove_file(&path);
    std::fs::create_dir_all(&dir).unwrap();

    let cfg = FMCfg {
        mid: MID,
        initial_count: 1,
        path: path.clone(),
        flush_duration: std::time::Duration::from_micros(100),
    };

    let mmap = FrozenMMap::<Meta>::new(cfg.clone()).unwrap();
    assert_eq!(mmap.slots(), 1);

    let (_, epoch) = mmap.write(0, |m| m.0 = 0x0A).unwrap();
    mmap.wait_for_durability(epoch).unwrap();

    let val = mmap.read(0, |m| m.0).unwrap();
    assert_eq!(val, 10);

    mmap.grow(1).unwrap();
    assert_eq!(mmap.slots(), 2);

    mmap.write(1, |m| m.0 = 0x0F).unwrap();
    drop(mmap); // syncs data

    // reopen & verify
    let reopened = FrozenMMap::<Meta>::new(cfg).unwrap();

    let v0 = reopened.read(0, |m| m.0).unwrap();
    let v1 = reopened.read(1, |m| m.0).unwrap();

    assert_eq!(v0, 0x0A);
    assert_eq!(v1, 0x0F);
}

§FrozenErr

FRes & FErr are custom implementation’s for result and error propagation.

To use the error module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["error"] }

§Hints

The hints module provides stable friendly implementations of likely and unlikely branch hints functions.

To use the hints module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["hints"] }

§Crc32

Implementation of CRC32C (Castagnoli polynomial) to compute a 32-bit cyclic redundancy check (CRC) using Castagnoli polynomial, intended for data integrity verification for torn writes and corruption detection

[!WARNING] We assume little-endian target architecture, as big-endian architectures are not supported

[!IMPORTANT] The generated 32-bit CRC is not cryptographically secure, it’s intended use only is for data integrity in IO ops

To use the crc32 module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["crc32"] }

Crc32C is available on following architectures,

ArchitectureSupport
aarch64✅
x86_64✅

Read the example below for usage details,

use frozen_core::crc32::Crc32C;

fn main() {
    let crc = Crc32C::new();

    let b0: [u8; 8] = *b"12345678";
    let b1: [u8; 8] = *b"ABCDEFGH";
    let b2: [u8; 8] = *b"abcdefgh";
    let b3: [u8; 8] = *b"zyxwvuts";

    assert_eq!(crc.crc(&b0), crc.crc(&b0));
    assert_eq!(crc.crc_2x([&b0, &b1]), crc.crc_2x([&b0, &b1]));
    assert_eq!(crc.crc_4x([&b0, &b1, &b2, &b3]), crc.crc_4x([&b0, &b1, &b2, &b3]));
}

§BPool

Lock-free buffer pool used for staging IO buffers

To use the bpool module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["bpool"] }

Read the example below for usage details,

use frozen_core::bpool::{BPBackend, BPCfg, BufPool};
use std::sync::Arc;
use std::thread;

fn main() {
    let pool = Arc::new(BufPool::new(BPCfg {
        mid: 0,
        chunk_size: 0x20,
        backend: BPBackend::Prealloc { capacity: 0x10 },
    }));

    let mut threads = Vec::new();
    for _ in 0..4 {
        let pool = pool.clone();

        threads.push(thread::spawn(move || {
            for _ in 0..0x80 {
                let alloc = pool.allocate(4).expect("allocation failed");
                assert_eq!(alloc.count, 4);
            }
        }));
    }

    for t in threads {
        t.join().expect("thread failed");
    }
}

§MPSCQ

A lock-free multi-producer single-consumer queue

To use the mpscq module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["mpscq"] }

Read the example below for usage details,

use frozen_core::mpscq::MPSCQueue;

fn main() {
    let queue = MPSCQueue::<usize>::default();

    queue.push(1usize);
    queue.push(2usize);
    queue.push(3usize);

    let batch = queue.drain();

    assert_eq!(batch.len(), 3);
    assert_eq!(batch, vec![3usize, 2usize, 1usize]);

    drop(batch); // values is dropped here
}

§FrozenPipe

An high throughput asynchronous IO pipeline for chunk based storage, it uses batches to write requests and flushes them in the background, while providing durability guarantees via epochs

To use the fpipe module, add it as a dependency in your Cargo.toml:

[dependencies]
frozen-core = { version = "0.0.11", default-features = false, features = ["fpipe"] }

Read the example below for usage details,

use frozen_core::bpool::{BPBackend, BPCfg, BufPool};
use frozen_core::ffile::{FFCfg, FrozenFile};
use frozen_core::fpipe::FrozenPipe;
use std::time::Duration;

fn main() {
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("tmp_pipe");

    let file = FrozenFile::new(FFCfg {
        path,
        mid: 0,
        chunk_size: 0x20,
        initial_chunk_amount: 4,
    })
    .unwrap();

    let pool = BufPool::new(BPCfg {
        mid: 0,
        chunk_size: 0x20,
        backend: BPBackend::Prealloc { capacity: 0x10 },
    });

    let pipe = FrozenPipe::new(file, pool, Duration::from_micros(0x3A)).unwrap();

    let buf = vec![1u8; 0x40];
    let epoch = pipe.write(&buf, 0).unwrap();

    pipe.wait_for_durability(epoch).unwrap();

    let read = pipe.read(0, 2).unwrap();
    assert_eq!(read, buf);
}

§Notes

[!IMPORTANT] frozen-core is primarily created for frozen-lab projects. External use is discouraged, but not prohibited, given you assume all the risks.

This project is licensed under the Apache-2.0 and MIT License. See the LICENSE-APACHE and LICENSE-MIT file for more details.

Contributions are welcome! Please feel free to submit a PR or open an issue if you have any feedback or suggestions.

Modules§

bpool
Lock-free buffer pool used for staging IO buffers
crc32
Implementation of CRC32C (Castagnoli polynomial) to compute a 32-bit cyclic redundancy check (CRC) using Castagnoli polynomial, intended for data integrity verification for torn writes and curruption detection
error
Custom implementations for Result<T> and Err<T>
ffile
Custom implementation of std::fs::File
fmmap
Custom implementation of mmap(2)
fpipe
An high throughput asynchronous IO pipeline for chunk based storage, it uses batches to write requests and flushes them in the background, while providing durability guarantees via epochs
hints
Implementation of branch preditor hints
mpscq
A lock-free multi-producer single-consumer queue