i2o2 0.4.0

A io_uring based IO executor for sync and async runtimes
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118

# I2o2

A tiny scheduler for executing IO calls with an io_uring executor which aims to be flexible while still being
simple to set up, in particular it supports _both_ 64 byte and 128 byte operating modes of io_uring which means
NVME pass through is supported.

This project is designed for [lnx](https://github.com/lnx-search/lnx) as a replacement to glommio's file system API
and is a relatively low-level API.

In particular, the system only lightly attempts to prevent you messing stuff up, hence why almost all calls
are unsafe. The only thing it really protects against buffers being dropped early.

## FAQ

#### What is the minimum supported kernel version

Version `5.15`+ is supported by I2o2, we may advance this requirement in future releases.
It is recommended to use kernel version `5.19`+ in order to support most features, but realistically,
the newer, the better (and faster :))

#### Should I use this in my own project?

Probably not unless you are sure you need the performance and are willing to invest the time into ensuring your
system is implemented safely.

#### Why not use glommio or tokio-uring?

In our use case, our main runtime is the [tokio](https://tokio.rs/) multithreaded scheduler, the only thing we
need a separate scheduler for is performing heavy file IO calls, which is why we reached for io_uring.

However, we do not need any of the task scheduling these other libraries provide and just adds another 
layer of complexity to an already complex storage system. 

So we have effectively stripped out all task scheduling logic and in effect, are left with just an actor for
completing IO calls without blocking our main runtime.

#### Do I need multiple schedulers to handle my load?

Probably not, at least I haven't had a situation where that is the case. In the no-op benchmarks which effectively
just tests the overhead of the scheduler and io_uring, we can reach upto 3 million ops per second with thousands concurrent 
workers all pushing data to the scheduler.

## Feature - Failpoint injection

If you enable the `fail` feature flag, i2o2 provides fail points via https://github.com/tikv/fail-rs to 
return custom/specific errors or status codes without having to trigger the actual error.

Function are targeted via `i2o2::fail::<function>`, for example you can set `FAILPOINTS=i2o2::fail::register_file=return(-4)`
to return the IO errno `4` aka `Interrupted`. 

See the `examples/fail_points` file for more demonstration.

## Example

```rust
use std::io;
use std::sync::Arc;

#[tokio::main]
async fn main() -> io::Result<()> {
    println!("creating our scheduler worker");

    // For most cases you can use `create_and_spawn` which uses a set of sane defaults.
    // The scheduler handle can be cheaply cloned and works in both sync and async contexts.
    let (thread_handle, scheduler_handle) = i2o2::create_and_spawn()?;

    // Now we can issue IO calls which would ordinarily be a syscall, like reading a file.
    // Now, this API is still unsafe, because you are responsible for ensuring the op is safe
    // to perform and buffers, etc... remain valid.

    let op = i2o2::opcode::Nop::new();

    // However, some utils are provided like the  `guard` parameter,  which will only be
    // dropped once the operation is complete and no longer needed by the kernel.
    let guard = Arc::new(());

    println!("submitting {op:?} to the scheduler");

    // We send the message to the scheduler asynchronously!
    let reply = unsafe {
        scheduler_handle
            .submit_async(op, Some(guard.clone()))
            .await
            .expect("submit op to scheduler")
    };
    println!("reply future created: {reply:?}");

    // We can synchronously or asynchronously wait for the reply, which will give us
    // back the result that the syscall equivalent of the operation would return.
    // I.e. `opcode::Write` would return the same value as `pwrite(2)`.
    let reply = reply.await;
    println!("our task completed with reply: {reply:?}");
    assert_eq!(reply, Ok(0));

    println!("shutting down scheduler");
    // The scheduler will shut down once all handles are dropped.
    // Any outstanding tasks will finish gracefully.
    drop(scheduler_handle);

    thread_handle
        .join()
        .expect("scheduler should never panic")?;

    println!("scheduler shutdown complete!");

    Ok(())
}
```

You can see more examples in the [example directory](/examples)


## Development

Anyone is welcome to contribute, just be aware I2o2 only wants to act as a slightly higher wrapper around io_uring
and just provide the async wrapper on top. We don't plan any support for making it more like a traditional runtime.

We also assume you have a very new kernel version for running tests, and by new I mean 6.1+ at least.