Crate i2o2

I2o2

A tiny scheduler for executing IO calls with an io_uring executor which aims to be flexible while still being simple to set up, in particular it supports both 64 byte and 128 byte operating modes of io_uring which means NVME pass through is supported.

This project is designed for lnx as a replacement to glommio’s file system API and is a relatively low-level API.

In particular, the system only lightly attempts to prevent you messing stuff up, hence why almost all calls are unsafe. The only thing it really protects against buffers being dropped early.

FAQ

What is the minimum supported kernel version

Version 5.15+ is supported by I2o2, we may advance this requirement in future releases. It is recommended to use kernel version 5.19+ in order to support most features, but realistically, the newer, the better (and faster :))

Should I use this in my own project?

Probably not unless you are sure you need the performance and are willing to invest the time into ensuring your system is implemented safely.

Why not use glommio or tokio-uring?

In our use case, our main runtime is the tokio multithreaded scheduler, the only thing we need a separate scheduler for is performing heavy file IO calls, which is why we reached for io_uring.

However, we do not need any of the task scheduling these other libraries provide and just adds another layer of complexity to an already complex storage system.

So we have effectively stripped out all task scheduling logic and in effect, are left with just an actor for completing IO calls without blocking our main runtime.

Do I need multiple schedulers to handle my load?

Probably not, at least I haven’t had a situation where that is the case. In the no-op benchmarks which effectively just tests the overhead of the scheduler and io_uring, we can reach upto 3 million ops per second with thousands concurrent workers all pushing data to the scheduler.

Example

use std::io;
use std::sync::Arc;

#[tokio::main]
async fn main() -> io::Result<()> {
    println!("creating our scheduler worker");

    // For most cases you can use `create_and_spawn` which uses a set of sane defaults.
    // The scheduler handle can be cheaply cloned and works in both sync and async contexts.
    let (thread_handle, scheduler_handle) = i2o2::create_and_spawn()?;

    // Now we can issue IO calls which would ordinarily be a syscall, like reading a file.
    // Now, this API is still unsafe, because you are responsible for ensuring the op is safe
    // to perform and buffers, etc... remain valid.

    let op = i2o2::opcode::Nop::new();

    // However, some utils are provided like the  `guard` parameter,  which will only be
    // dropped once the operation is complete and no longer needed by the kernel.
    let guard = Arc::new(());

    println!("submitting {op:?} to the scheduler");

    // We send the message to the scheduler asynchronously!
    let reply = unsafe {
        scheduler_handle
            .submit_async(op, Some(guard.clone()))
            .await
            .expect("submit op to scheduler")
    };
    println!("reply future created: {reply:?}");

    // We can synchronously or asynchronously wait for the reply, which will give us
    // back the result that the syscall equivalent of the operation would return.
    // I.e. `opcode::Write` would return the same value as `pwrite(2)`.
    let reply = reply.await;
    println!("our task completed with reply: {reply:?}");
    assert_eq!(reply, Ok(0));

    println!("shutting down scheduler");
    // The scheduler will shut down once all handles are dropped.
    // Any outstanding tasks will finish gracefully.
    drop(scheduler_handle);

    thread_handle
        .join()
        .expect("scheduler should never panic")?;

    println!("scheduler shutdown complete!");

    Ok(())
}

You can see more examples in the example directory

Development

Anyone is welcome to contribute, just be aware I2o2 only wants to act as a slightly higher wrapper around io_uring and just provide the async wrapper on top. We don’t plan any support for making it more like a traditional runtime.

We also assume you have a very new kernel version for running tests, and by new I mean 6.1+ at least.

Re-exports

pub use self::opcode::types;

Modules

opcode

Structs

CpuSet

The cpu set restricts what CPU cores the thread can run on.

I2o2Builder

A set of configuration options for customising the I2o2Scheduler scheduler.

I2o2Handle

The I2o2Handle allows you to interact with the I2o2Scheduler and submit IO events to it.

I2o2Scheduler

The I2o2Scheduler runs an io_uring ring in the current thread and submits IO events from the handle into the ring.

ReplyReceiver

Wait for a reply for an IO result.

SchedulerClosed

The scheduler has shutdown and is no longer accepting events.

Enums

RegisterError

An error that prevent reregistration of a resource.

TryGetResultError

Attempt to get the result of the operation.

Functions

builder

Create a new I2o2Scheduler and I2o2Handle pair backed by io_uring with a custom configuration.

create_and_spawn

Create a new I2o2Scheduler and I2o2Handle pair backed by io_uring and spawn the scheduler in a background worker thread.

create_for_current_thread

Create a new I2o2Scheduler and I2o2Handle pair backed by io_uring.

Type Aliases

DynamicGuard

A guard type that can be any object.

SubmitResult

A submission result for the scheduler.