fjall 2.4.2

LSM-based key-value storage engine
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190

<p align="center">
  <img src="/kawaii.png" height="200">
</p>

<p align="center">
  <a href="https://github.com/fjall-rs/fjall/actions/workflows/test.yml">
    <img src="https://github.com/fjall-rs/fjall/actions/workflows/test.yml/badge.svg" alt="CI" />
  </a>
  <a href="https://docs.rs/fjall">
    <img src="https://img.shields.io/docsrs/fjall?color=green" alt="docs.rs" />
  </a>
  <a href="https://crates.io/crates/fjall">
    <img src="https://img.shields.io/crates/v/fjall?color=blue" alt="Crates.io" />
  </a>
  <img src="https://img.shields.io/badge/MSRV-1.74.0-blue" alt="MSRV" />
  <a href="https://deps.rs/repo/github/fjall-rs/fjall">
    <img src="https://deps.rs/repo/github/fjall-rs/fjall/status.svg" alt="dependency status" />
  </a>
</p>

<p align="center">
  <a href="https://discord.com/invite/HvYGp4NFFk">
    <img src="https://img.shields.io/discord/1240426554111164486" alt="Discord" />
  </a>
  <a href="https://bsky.app/profile/fjallrs.bsky.social">
    <img src="https://img.shields.io/badge/bluesky-blue" alt="Bluesky" />
  </a>
</p>

*Fjall* is an LSM-based embeddable key-value storage engine written in Rust.
It features:

- Thread-safe BTreeMap-like API
- 100% safe & stable Rust
- Range & prefix searching with forward and reverse iteration
- Partitions (a.k.a. column families) with cross-partition atomic semantics
- Built-in compression (default = LZ4)
- Serializable transactions (optional)
- Key-value separation for large blob use cases (optional)
- Automatic background maintenance

It is not:

- a standalone server
- a relational database
- a wide-column database: it has no built-in notion of columns

Keys are limited to 65536 bytes, values are limited to 2^32 bytes.
As is normal with any kind of storage engine, larger keys and values have a bigger performance impact.

Like any typical key-value store, keys are stored in lexicographic order.
If you are storing integer keys (e.g. timeseries data), you should use the big endian form to adhere to locality.

## Basic usage

```bash
cargo add fjall
```

```rust
use fjall::{Config, PersistMode, Keyspace, PartitionCreateOptions};

// A keyspace is a database, which may contain multiple collections ("partitions")
// You should probably only use a single keyspace for your application
//
let keyspace = Config::new(folder).open()?; // or open_transactional for transactional semantics

// Each partition is its own physical LSM-tree
let items = keyspace.open_partition("my_items", PartitionCreateOptions::default())?;

// Write some data
items.insert("a", "hello")?;

// And retrieve it
let bytes = items.get("a")?;

// Or remove it again
items.remove("a")?;

// Search by prefix
for kv in items.prefix("prefix") {
  // ...
}

// Search by range
for kv in items.range("a"..="z") {
  // ...
}

// Iterators implement DoubleEndedIterator, so you can search backwards, too!
for kv in items.prefix("prefix").rev() {
  // ...
}

// Sync the journal to disk to make sure data is definitely durable
// When the keyspace is dropped, it will try to persist with `PersistMode::SyncAll` as well
keyspace.persist(PersistMode::SyncAll)?;
```

## Durability

To support different kinds of workloads, Fjall is agnostic about the type of durability
your application needs.
After writing data (`insert`, `remove` or committing a write batch/transaction), you can choose to call [`Keyspace::persist`](https://docs.rs/fjall/latest/fjall/struct.Keyspace.html#method.persist) which takes a [`PersistMode`](https://docs.rs/fjall/latest/fjall/enum.PersistMode.html) parameter.
By default, any operation will flush to OS buffers, but **not** to disk.
This is in line with RocksDB's default durability.
Also, when dropped, the keyspace will try to persist the journal *to disk* synchronously.

## Multithreading, Async and Multiprocess

> !!! A single keyspace may **not** be loaded in parallel from separate *processes*.

However, Fjall is internally synchronized for multi-threaded access, so you can clone around the `Keyspace` and `Partition`s as needed, without needing to lock yourself.

For an async example, see the [`tokio`](https://github.com/fjall-rs/fjall/tree/main/examples/tokio) example.

## Feature flags

### lz4

Allows using `LZ4` compression, powered by [`lz4_flex`](https://github.com/PSeitz/lz4_flex).

*Enabled by default.*

### miniz

Allows using `DEFLATE/zlib` compression, powered by [`miniz_oxide`](https://github.com/Frommi/miniz_oxide).

*Disabled by default.*

### single_writer_tx

Allows opening a transactional Keyspace for single-writer (serialized) transactions, allowing RYOW (read-your-own-write), fetch-and-update and other atomic operations.

*Enabled by default.*

### ssi_tx

Allows opening a transactional Keyspace for multi-writer, serializable transactions, allowing RYOW (read-your-own-write), fetch-and-update and other atomic operations.
Conflict checking is done using optimistic concurrency control.

*Disabled by default.*

### bytes

Uses [`bytes`](https://github.com/tokio-rs/bytes) as the underlying `Slice` type.

*Disabled by default.*

### bloom *[deprecated]*

Uses bloom filters to reduce disk I/O when serving point reads, but increases memory usage.

*Enabled by default.*

> Will be removed in the future.
> If you are absolutely, 100% sure you do not need bloom filters: they will be togglable on a per-partition basis.

## Stable disk format

The disk format is stable as of 1.0.0.

2.0.0 uses a new disk format and needs a manual format migration.

Future breaking changes will result in a major version bump and a migration path.

For the underlying LSM-tree implementation, see: <https://crates.io/crates/lsm-tree>.

## Examples

[See here](https://github.com/fjall-rs/fjall/tree/main/examples) for practical examples.

And checkout [`Smoltable`](https://github.com/marvin-j97/smoltable), a standalone Bigtable-inspired toy wide-column database using `fjall` as its storage engine.

## Contributing

How can you help?

- [Ask a question](https://github.com/fjall-rs/fjall/discussions/new?category=q-a)
  - or join the Discord server: [https://discord.com/invite/HvYGp4NFFk](https://discord.com/invite/HvYGp4NFFk)
- [Post benchmarks and things you created](https://github.com/fjall-rs/fjall/discussions/new?category=show-and-tell)
- [Open a PR](https://github.com/fjall-rs/fjall/compare), 
  - [See open issues to pick up here](https://github.com/search?q=org%3Afjall-rs+label%3A%22help+wanted%22+state%3Aopen+&type=issues)
- [Open an issue](https://github.com/fjall-rs/fjall/issues/new) (bug report, weirdness)

## License

All source code is licensed under MIT OR Apache-2.0.

All contributions are to be licensed as MIT OR Apache-2.0.