# What is it?
This crate implements a solution to efficiently handle several record logs.
Each recordlog has its own "local" notion of position.
It is possible to truncate each of the queues individually.
# Goals
- be durable, offer some flexibility on `fsync` strategies.
- offer a way to truncate a queue after a specific position
- handle an arbitrary number of queues
- have limited IO
- be fast
- offer the possibility to implement push back
```rust
pub struct MultiRecordLog {
pub async fn create_queue(&mut self, queue: &str) -> Result<(), CreateQueueError>;
pub async fn delete_queue(&mut self, queue: &str) -> Result<(), DeleteQueueError>;
pub fn queue_exists(&self, queue: &str) -> bool;
pub fn list_queues(&self) -> impl Iterator<Item = &str> {
pub async fn append_record(
&mut self,
queue: &str,
position_opt: Option<u64>,
payload: &[u8],
);
pub async fn truncate(&mut self, queue: &str, position: u64) -> Result<(), TruncateError>;
pub fn range<R>(
&self,
queue: &str,
range: R,
) -> Option<impl Iterator<Item = (u64, &[u8])> + '_>;
}
```
# Non-goals
This is not Kafka. That recordlog is designed for a "small amount of data".
All retained data can fit in RAM.
In the context of quickwit, that queue is used in the PushAPI and is meant to contain
1 worth of data. (At 60MB/s, means 3.6 GB of RAM)
Reading the recordlog files only happens on startup.
High-performance when reading the recordlog files is not a goal.
Writing fast on the other hand is important.
# Implementation details.
`mrecordlog` is multiplexing several independent queues into the same record log.
This approach has the merit of limiting the number of file descriptor necessary,
and more importantly, to limit the number of `fsync`.
It also offers the possibility to truncate the queue for a given record log.
The actual deletion of the data happens when a file only contains deleted records.
Then, and only then, the entire file is deleted.
That recordlog emits a new file every 1GB.
A recordlog file is deleted, once all queues have been truncated after the
last record of a of a file.
There are no compaction logic.
# TODO
- add backpressure.
- add fsync policy
- better testing.
- non auto-inc position
- less Arc