- an embedded kv store
- a construction kit for stateful systems
- ordered map API similar to a Rust
- fully atomic single-key operations, supports CAS
- zero-copy reads
- merge operators
- forward and reverse iterators
- a monotonic ID generator capable of giving out 75-125+ million unique IDs per second, never double allocating even in the presence of crashes
- zstd compression (use the zstd build feature)
- cpu-scalable lock-free implementation
- SSD-optimized log-structured storage
People face unnecessary hardship when working with existing embedded databases. They tend to have sharp performance trade-offs, are difficult to tune, have unclear consistency guarantees, and are generally inflexible. Facebook uses distributed machine learning to find configurations that achieve great performance for specific workloads on rocksdb. Most engineers don't have access to that kind of infrastructure. We would like to build sled so that it can be optimized using simple local methods, with as little user input as possible, and in many cases exceed the performance of popular systems today.
This is how we aim to improve the situation:
- don't make the user think. the interface should be obvious.
- don't surprise users with performance traps.
- don't wake up operators. bring reliability techniques from academia into real-world practice.
- don't use so much electricity. our data structures should play to modern hardware's strengths.
sled is written by people with experience designing, building, testing, and operating databases at high scales. we think the situation can be improved.
Building a database takes years. Designers of databases make bets about target usage and hardware. Here are the trends that we see, which we want to optimize the experience around:
- more cores on servers, spanning sockets and numa domains
- the vast majority of content consumption and generation happening on phones
- compute migrating to the edge, into CDNs
- conflict-free and OT-based replication techniques at the edge
- strongly-consistent replication techniques within and between datacenters
- event-driven architectures which benefit heavily from subscriber/watch semantics
Over the years that sled development has been active, some practices have been collected that have helped to reduce risks throughout the codebase.
This page documents some limitations that sled imposes on users.
Merge operators are an extremely powerful tool for use in embedded kv stores. They allow users to specify custom logic for combining multiple versions of a value into one.
As of sled
Here's a look at where sled is at, and where it's going architecturally. The system is very much under active development, and we have a ways to go. If specific areas are interesting to you, I'd love to work together! If your business has a need for particular items below, you can fund development of particular features.
We believe operators of stateful systems should get as much sleep as they want. We take testing seriously, and we take pains to avoid the pesticide paradox wherever possible.