[](https://crates.io/crates/wedb_embed)
[](https://docs.rs/wedb_embed)
---
<a id="en"></a>
<h1 id="wedb_embed_en">wedb_embed</h1>
Embedded database engine providing Redis-compatible data structures and APIs, built on the [fjall](https://github.com/fjall-rs/fjall) LSM-Tree storage engine.
<p align="center">
<img src="https://fastly.jsdelivr.net/gh/webc-fs/-@l6/t47NzKcVXrhpFuFM1cDg.svg" alt="wedb_embed vs Redis Performance & Resource Comparison" width="100%">
<br>
<sub><b>Benchmark Environment</b>: CPU: Apple M2 Max (12 cores) ｜ Memory: 64.0 GB ｜ OS: macOS 26.5.1 (Darwin 25.5.0) ｜ Rust: 1.98.0 (88d9e12ae 2026-08-18) ｜ Redis: v8.10.1</sub>
</p>
---
## Why an Embedded Redis Engine
- [Why an Embedded Redis Engine](#why-an-embedded-redis-engine)
- [Quickstart](#quickstart)
- [Installation](#installation)
- [Basic Usage](#basic-usage)
- [Performance & Resource Comparison](#performance-resource-comparison)
- [macOS (Apple M2 Max)](#macos-apple-m2-max)
- [Hardware & Test Environment](#hardware-test-environment)
- [Physical Footprint & Memory Benchmark (5GB Dataset Scale)](#physical-footprint-memory-benchmark-5gb-dataset-scale)
- [wedb_embed vs Redis Core Command Benchmark](#wedb_embed-vs-redis-core-command-benchmark)
- [wedb_embed Performance Regression](#wedb_embed-performance-regression)
- [Storage Architecture & Encoding Design](#storage-architecture-encoding-design)
- [Dual-Track Storage Partitioning & Zero-Byte Prefix](#dual-track-storage-partitioning-zero-byte-prefix)
- [Order-Preserving Prefix Varint (OPPV)](#order-preserving-prefix-varint-oppv)
- [Numerical Tenant Mapping & Atomic Renaming](#numerical-tenant-mapping-atomic-renaming)
- [Memory-Efficient Streaming Iterators](#memory-efficient-streaming-iterators)
- [Runtime Architecture & Threading Model](#runtime-architecture-threading-model)
- [Thread-per-Core Architecture Design](#thread-per-core-architecture-design)
- [Pitfalls of Multi-Threaded Work-Stealing Runtimes](#pitfalls-of-multi-threaded-work-stealing-runtimes)
- [Multi-Tenant & Multi-DB Isolation](#multi-tenant-multi-db-isolation)
- [Data Structures & Operations](#data-structures-operations)
- [Initialization & Configuration](#initialization-configuration)
- [Key-Value & Strings](#key-value-strings)
- [Hash Map & Field-Level TTL](#hash-map-field-level-ttl)
- [List](#list)
- [Set](#set)
- [Sorted Set](#sorted-set)
- [Bitmap & Bitfield](#bitmap-bitfield)
- [JSON & JSONPath](#json-jsonpath)
- [Bloom & Cuckoo Filters](#bloom-cuckoo-filters)
- [TimeSeries & Aggregations](#timeseries-aggregations)
- [Geospatial](#geospatial)
- [HyperLogLog](#hyperloglog)
- [T-Digest](#t-digest)
- [SortedInt](#sortedint)
- [Streams & Consumer Groups](#streams-consumer-groups)
- [Full-Text Search & Vector Retrieval](#full-text-search-vector-retrieval)
- [Tech Stack](#tech-stack)
In backend services, CLI tools, edge computing nodes, and desktop applications, developers frequently require composite data structures such as hash maps, sorted sets, message streams, bitmaps, and time-series collections. The conventional approach involves running an external Redis daemon and communicating over network or Unix domain sockets. In standalone and embedded environments, this architecture incurs specific performance and resource constraints:
1. **IPC and Protocol Serialization Overhead**: Every read or write operation traverses socket buffers, triggers OS context switches, and requires RESP protocol encoding and decoding within an event loop. Even on `localhost`, round-trip latency typically remains in the 20–50 microsecond range while consuming CPU cycles.
2. **RAM Costs and Memory Limits**: Redis keeps all datasets and pointer-heavy metadata resident in physical RAM. As dataset volume expands to tens of gigabytes, memory hardware costs escalate and remain strictly bounded by host RAM capacity. Background AOF/RDB persistence can further double memory usage via Copy-On-Write mechanisms.
3. **Deployment and Operational Overhead**: Managing external daemon processes requires process supervisors, port allocation, configuration syncing, and health monitoring.
`wedb_embed` integrates the storage engine directly into the application process, transforming the data access path:
- **In-Process Direct Invocation**: Data operations execute directly via Rust function calls in memory, eliminating socket I/O, syscalls, and inter-process context switches. P95 latency for core commands is reduced from tens of microseconds to nanosecond and microsecond ranges.
- **LSM-Tree Disk Persistence with Tiered Storage**: Built on an LSM-Tree engine with LZ4 block compression, hot data resides in memory buffers while full datasets persist compressed on disk. In a 2GB structured dataset benchmark, resident memory (RSS) dropped from 1951 MB (Redis) to 234 MB (an 88% reduction), while disk footprint was reduced by 38%.
- **Comprehensive Redis Data Models**: Provides 16 composite data models built atop the key-value core (String, Hash with field-level TTL, List, Set, ZSet, Bitmap, JSON, Bloom/Cuckoo Filters, TimeSeries, Geo, HyperLogLog, TDigest, SortedInt, Stream, Full-Text Search, and HNSW Vector Retrieval).
- **Multi-Tenant and Multi-DB Isolation**: Supports up to $2^{64}$ isolated tenants and databases. Tenant renaming updates only metadata mappings in $O(1)$ time without data rewrites.
- **Crash Consistency**: Relies on Write-Ahead Logging (WAL) and cross-keyspace atomic write batches to maintain data integrity across crashes.
---
## Quickstart
### Installation
```bash
cargo add wedb_embed
```
### Basic Usage
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
// 1. Open database
let db = WeDb::open("./data/quickstart_db", [])?;
// 2. String operations
db.set(b"site", b"webc.site", &[])?;
let val = db.get(b"site")?;
assert_eq!(val.as_deref(), Some(&b"webc.site"[..]));
// 3. Hash operations
db.hset(b"user:1", &[(b"name", b"Alice"), (b"age", b"20")])?;
let age = db.hget(b"user:1", b"age")?;
assert_eq!(age.as_deref(), Some(&b"20"[..]));
// 4. Sorted set operations
db.zadd(b"rank", &[(100.0, b"player1"), (200.0, b"player2")], &[])?;
let top = db.zrange(b"rank", 0, 10)?;
assert_eq!(top.len(), 2);
Ok(())
}
```
[Click here for more examples](../examples)
---
- [Why an Embedded Redis Engine](#why-an-embedded-redis-engine)
- [Quickstart](#quickstart)
- [Installation](#installation)
- [Basic Usage](#basic-usage)
- [Performance & Resource Comparison](#performance--resource-comparison)
- [Storage Architecture & Encoding Design](#storage-architecture--encoding-design)
- [Dual-Track Storage Partitioning & Zero-Byte Prefix](#dual-track-storage-partitioning--zero-byte-prefix)
- [Order-Preserving Prefix Varint (OPPV)](#order-preserving-prefix-varint-oppv)
- [Numerical Tenant Mapping & Atomic Renaming](#numerical-tenant-mapping--atomic-renaming)
- [Memory-Efficient Streaming Iterators](#memory-efficient-streaming-iterators)
- [Runtime Architecture & Threading Model](#runtime-architecture--threading-model)
- [Thread-per-Core Architecture Design](#thread-per-core-architecture-design)
- [Pitfalls of Multi-Threaded Work-Stealing Runtimes](#pitfalls-of-multi-threaded-work-stealing-runtimes)
- [Multi-Tenant & Multi-DB Isolation](#multi-tenant--multi-db-isolation)
- [Data Structures & Operations](#data-structures--operations)
- [Initialization & Configuration](#initialization--configuration)
- [Key-Value & Strings](#key-value--strings)
- [Hash Map & Field-Level TTL](#hash-map--field-level-ttl)
- [List](#list)
- [Set](#set)
- [Sorted Set](#sorted-set)
- [Bitmap & Bitfield](#bitmap--bitfield)
- [JSON & JSONPath](#json--jsonpath)
- [Bloom & Cuckoo Filters](#bloom--cuckoo-filters)
- [TimeSeries & Aggregations](#timeseries--aggregations)
- [Geospatial](#geospatial)
- [HyperLogLog](#hyperloglog)
- [T-Digest](#t-digest)
- [SortedInt](#sortedint)
- [Streams & Consumer Groups](#streams--consumer-groups)
- [Full-Text Search & Vector Retrieval](#full-text-search--vector-retrieval)
- [Tech Stack](#tech-stack)
---
## Performance & Resource Comparison
### macOS (Apple M2 Max)
#### Hardware & Test Environment
CPU: Apple M2 Max (12 cores)<br>
Memory: 64.0 GB<br>
OS: macOS 26.5.1 (Darwin 25.5.0)<br>
Rust: 1.98.0 (88d9e12ae 2026-08-18)<br>
Redis: v8.10.1
#### Physical Footprint & Memory Benchmark (5GB Dataset Scale)
| **Dataset Scale** | 5,000,000 Structured Items | 5,000,000 Structured Items | All 14 Data Formats |
| **Raw Uncompressed Payload** | 4377 MB | 4377 MB | Structured Payload |
| **Physical Disk Footprint** | **4791 MB** | **7791 MB** | **Saves 39%** |
| **Resident Memory (RSS)** | **508 MB** | **4918 MB** | **Saves 90%** |
#### wedb_embed vs Redis Core Command Benchmark
| `SET` | 9.1 us | 47.3 us | **5.2x** |
| `GET` | 0.83 us | 41.8 us | **50.2x** |
| `MSET` | 52.2 us | 54.6 us | **1.0x** |
| `MGET` | 2.4 us | 43.3 us | **17.9x** |
| `INCRBY` | 0.58 us | 48.1 us | **82.8x** |
| `DECRBY` | 0.58 us | 45.9 us | **79.8x** |
| `APPEND` | 0.78 us | 49.4 us | **63.4x** |
| `STRLEN` | 0.24 us | 40.9 us | **168.1x** |
| `GETDEL` | 10.1 us | 94.5 us | **9.4x** |
| `GETRANGE` | 0.25 us | 42.8 us | **171.3x** |
| `SETRANGE` | 0.59 us | 45.0 us | **75.9x** |
| `HSET` | 3.0 us | 48.4 us | **16.0x** |
| `HGET` | 0.70 us | 47.0 us | **67.2x** |
| `HMGET` | 2.7 us | 45.3 us | **17.0x** |
| `HEXISTS` | 0.64 us | 45.7 us | **71.7x** |
| `HLEN` | 0.47 us | 43.0 us | **90.7x** |
| `HDEL` | 4.9 us | 43.0 us | **8.7x** |
| `HGETALL` | 3.2 us | 44.1 us | **13.9x** |
| `HKEYS` | 3.1 us | 45.8 us | **14.7x** |
| `HVALS` | 3.1 us | 42.8 us | **13.7x** |
| `HINCRBY` | 3.0 us | 45.6 us | **15.3x** |
| `LPUSH` | 2.9 us | 44.1 us | **15.0x** |
| `RPUSH` | 3.4 us | 43.8 us | **13.0x** |
| `LPOP` | 3.0 us | 44.7 us | **14.7x** |
| `RPOP` | 3.2 us | 43.5 us | **13.8x** |
| `LLEN` | 0.46 us | 40.8 us | **89.1x** |
| `LRANGE` | 2.6 us | 42.4 us | **16.2x** |
| `LINDEX` | 0.67 us | 41.0 us | **61.3x** |
| `LSET` | 2.2 us | 44.0 us | **19.9x** |
| `LREM` | 14.0 us | 93.9 us | **6.7x** |
| `LTRIM` | 2.8 us | 43.1 us | **15.4x** |
| `SADD` | 2.3 us | 44.9 us | **19.2x** |
| `SREM` | 4.8 us | 46.5 us | **9.7x** |
| `SISMEMBER` | 0.65 us | 41.3 us | **63.9x** |
| `SCARD` | 0.48 us | 41.1 us | **86.3x** |
| `SMEMBERS` | 3.2 us | 41.8 us | **13.0x** |
| `SPOP` | 9.2 us | 88.4 us | **9.6x** |
| `SRANDMEMBER` | 3.3 us | 40.9 us | **12.4x** |
| `ZADD` | 3.7 us | 43.9 us | **11.8x** |
| `ZSCORE` | 0.72 us | 44.3 us | **61.3x** |
| `ZRANGE` | 3.7 us | 45.8 us | **12.2x** |
| `ZCARD` | 0.49 us | 40.8 us | **82.5x** |
| `ZCOUNT` | 3.2 us | 41.6 us | **13.1x** |
| `ZINCRBY` | 3.9 us | 44.6 us | **11.4x** |
| `ZRANK` | 3.6 us | 41.7 us | **11.7x** |
| `ZREVRANGE` | 5.3 us | 46.0 us | **8.7x** |
| `ZPOPMIN` | 9.0 us | 97.0 us | **10.8x** |
| `ZREM` | 5.1 us | 43.0 us | **8.4x** |
| `SETBIT` | 9.5 us | 55.4 us | **5.8x** |
| `GETBIT` | 0.42 us | 47.5 us | **112.7x** |
| `BITCOUNT` | 0.44 us | 46.0 us | **104.3x** |
| `BITPOS` | 0.45 us | 50.0 us | **111.0x** |
| `PFADD` | 4.5 us | 44.9 us | **10.0x** |
| `PFCOUNT` | 32.3 us | 41.5 us | **1.3x** |
| `GEOADD` | 4.0 us | 46.3 us | **11.6x** |
| `GEODIST` | 0.86 us | 41.7 us | **48.6x** |
| `GEOPOS` | 0.65 us | 41.3 us | **63.2x** |
| `GEOHASH` | 0.70 us | 41.1 us | **58.9x** |
| `XADD` | 2.7 us | 44.9 us | **16.8x** |
| `XLEN` | 0.45 us | 40.7 us | **90.3x** |
| `XRANGE` | 3.8 us | 47.8 us | **12.5x** |
| `XREAD` | 3.7 us | 49.1 us | **13.1x** |
| `XDEL` | 5.3 us | 88.3 us | **16.6x** |
| `DEL` | 4.1 us | 35.6 us | **8.6x** |
| `EXISTS` | 0.20 us | 35.1 us | **172.9x** |
| `EXPIRE` | 2.0 us | 42.0 us | **21.0x** |
| `TTL` | 0.25 us | 42.0 us | **168.1x** |
| `JSON.SET` | 4.1 us | 45.8 us | **11.1x** |
| `JSON.GET` | 1.3 us | 46.4 us | **36.6x** |
| `JSON.DEL` | 11.3 us | 101.5 us | **9.0x** |
| `JSON.NUMINCRBY` | 4.2 us | 48.2 us | **11.4x** |
| `JSON.ARRLEN` | 1.1 us | 46.4 us | **42.1x** |
| `JSON.TYPE` | 1.1 us | 42.5 us | **37.8x** |
| `BF.ADD` | 20.6 us | 48.2 us | **2.3x** |
| `BF.EXISTS` | 0.63 us | 37.4 us | **59.7x** |
| `BF.INFO` | 0.44 us | 48.6 us | **110.8x** |
| `CF.ADD` | 4.1 us | 39.7 us | **9.7x** |
| `CF.EXISTS` | 0.66 us | 35.5 us | **53.6x** |
| `CF.DEL` | 7.6 us | 84.7 us | **11.1x** |
| `TDIGEST.ADD` | 3.3 us | 41.2 us | **12.5x** |
| `TDIGEST.QUANTILE` | 0.83 us | 41.4 us | **49.7x** |
| `TDIGEST.BYRANK` | 0.83 us | 43.4 us | **52.1x** |
| `TDIGEST.CDF` | 1.0 us | 45.4 us | **43.6x** |
| `TS.ADD` | 6.5 us | 44.8 us | **6.9x** |
| `TS.GET` | 2.2 us | 43.4 us | **19.6x** |
| `TS.RANGE` | 10.6 us | 70.6 us | **6.7x** |
| `TS.INCRBY` | 9.8 us | 45.2 us | **4.6x** |
| `FT.SEARCH` | 19.8 us | 77.2 us | **3.9x** |
| `FT.TAG` | 19.6 us | 66.4 us | **3.4x** |
| `VECTOR.KNN` | 2.2 us | 64.8 us | **28.8x** |
#### wedb_embed Performance Regression
| `BITCOUNT` | 437.2 ns | 237 万/s | 0% |
| `BITPOS` | 1.135 µs | 96 万/s | 0% |
| `GETBIT` | 468.4 ns | 218 万/s | 0% |
| `SETBIT` | 117.7 µs | 9 万/s | 0% |
| `BF.ADD` | 50.79 µs | 3 万/s | 0% |
| `BF.EXISTS` | 713.2 ns | 150 万/s | 0% |
| `BF.INFO` | 432 ns | 233 万/s | 0% |
| `CF.ADD` | 55.83 µs | 16 万/s | 0% |
| `CF.DEL` | 34.16 µs | 9 万/s | 0% |
| `CF.EXISTS` | 812.2 ns | 138 万/s | 0% |
| `BATCH_COMMIT` | 34.66 µs | 16 万/s | 0% |
| `DEL` | 157.8 µs | 5 万/s | 0% |
| `EXISTS` | 170.2 ns | 601 万/s | 0% |
| `EXPIRE` | 6.041 µs | 29 万/s | 0% |
| `NAMESPACE` | 11.45 µs | 19 万/s | 0% |
| `TTL` | 227.5 ns | 450 万/s | 0% |
| `GEOADD` | 227.4 µs | 16 万/s | 0% |
| `GEODIST` | 1.244 µs | 84 万/s | 0% |
| `GEOHASH` | 728.9 ns | 139 万/s | 0% |
| `GEOPOS` | 984 ns | 123 万/s | 0% |
| `HDEL` | 5.208 µs | 26 万/s | 0% |
| `HEXISTS` | 640.3 ns | 159 万/s | 0% |
| `HGET` | 728.8 ns | 142 万/s | 0% |
| `HGETALL` | 30.7 µs | 27 万/s | 0% |
| `HINCRBY` | 172.7 µs | 19 万/s | 0% |
| `HKEYS` | 33.04 µs | 28 万/s | 0% |
| `HLEN` | 471 ns | 216 万/s | 0% |
| `HMGET` | 2.728 µs | 40 万/s | 0% |
| `HSET` | 176.5 µs | 19 万/s | 0% |
| `HVALS` | 25.29 µs | 30 万/s | 0% |
| `PFADD` | 191.3 µs | 6 万/s | 0% |
| `PFCOUNT` | 41.04 µs | 3 万/s | 0% |
| `PFMERGE` | 40.95 µs | 7 万/s | 0% |
| `JSON.ARRLEN` | 43.45 µs | 71 万/s | 0% |
| `JSON.DEL` | 12.83 µs | 11 万/s | 0% |
| `JSON.GET` | 26.49 µs | 63 万/s | 0% |
| `JSON.NUMINCRBY` | 48.29 µs | 18 万/s | 0% |
| `JSON.SET` | 64.58 µs | 13 万/s | 0% |
| `JSON.TYPE` | 1.676 µs | 87 万/s | 0% |
| `LINDEX` | 833 ns | 135 万/s | 0% |
| `LLEN` | 484 ns | 211 万/s | 0% |
| `LPOP` | 21.41 µs | 6 万/s | 0% |
| `LPUSH` | 191.9 µs | 20 万/s | 0% |
| `LRANGE` | 4.582 µs | 32 万/s | 0% |
| `LREM` | 224 µs | 5 万/s | 0% |
| `LSET` | 4.083 µs | 32 万/s | 0% |
| `LTRIM` | 7.29 µs | 33 万/s | 0% |
| `RPOP` | 8.416 µs | 13 万/s | 0% |
| `RPUSH` | 132.6 µs | 19 万/s | 0% |
| `FT.SEARCH` | 42.2 µs | 4 万/s | 0% |
| `FT.TAG` | 27.66 µs | 4 万/s | 0% |
| `SADD` | 150.6 µs | 16 万/s | 0% |
| `SCARD` | 520.4 ns | 202 万/s | 0% |
| `SISMEMBER` | 765.3 ns | 138 万/s | 0% |
| `SMEMBERS` | 6.291 µs | 30 万/s | 0% |
| `SPOP` | 17.95 µs | 10 万/s | 0% |
| `SRANDMEMBER` | 6.249 µs | 22 万/s | 0% |
| `SREM` | 4.916 µs | 25 万/s | 0% |
| `SI.ADD` | 187 µs | 19 万/s | 0% |
| `SI.CARD` | 585.6 ns | 210 万/s | 0% |
| `SI.EXISTS` | 822.5 ns | 123 万/s | 0% |
| `SI.RANGE` | 17.49 µs | 15 万/s | 0% |
| `SI.REM` | 158.2 µs | 6 万/s | 0% |
| `APPEND` | 609 ns | 175 万/s | 0% |
| `DECRBY` | 666.4 ns | 166 万/s | 0% |
| `GET` | 260.1 ns | 418 万/s | 0% |
| `GETDEL` | 180.7 µs | 7 万/s | 0% |
| `GETRANGE` | 372.1 ns | 500 万/s | 0% |
| `INCRBY` | 775.7 ns | 147 万/s | 0% |
| `MGET` | 2.666 µs | 42 万/s | 0% |
| `MSET` | 172.2 µs | 2 万/s | 0% |
| `SET` | 121 µs | 8 万/s | 0% |
| `SETRANGE` | 872.1 ns | 195 万/s | 0% |
| `STRLEN` | 197.6 ns | 516 万/s | 0% |
| `XADD` | 176.1 µs | 21 万/s | 0% |
| `XDEL` | 169.2 µs | 10 万/s | 0% |
| `XLEN` | 486.7 ns | 207 万/s | 0% |
| `XRANGE` | 12.33 µs | 11 万/s | 0% |
| `XREAD` | 17.29 µs | 11 万/s | 0% |
| `TDIGEST.ADD` | 9.791 µs | 19 万/s | 0% |
| `TDIGEST.BYRANK` | 13.83 µs | 89 万/s | 0% |
| `TDIGEST.CDF` | 10.91 µs | 80 万/s | 0% |
| `TDIGEST.QUANTILE` | 12.49 µs | 83 万/s | 0% |
| `TS.ADD` | 66.62 µs | 12 万/s | 0% |
| `TS.GET` | 5.291 µs | 43 万/s | 0% |
| `TS.INCRBY` | 53.62 µs | 8 万/s | 0% |
| `TS.RANGE` | 53.87 µs | 3 万/s | 0% |
| `VECTOR.KNN` | 8.165 µs | 34 万/s | 0% |
| `ZADD` | 169.9 µs | 18 万/s | 0% |
| `ZCARD` | 538.7 ns | 226 万/s | 0% |
| `ZCOUNT` | 5.874 µs | 31 万/s | 0% |
| `ZINCRBY` | 146.8 µs | 19 万/s | 0% |
| `ZPOPMIN` | 14.79 µs | 13 万/s | 0% |
| `ZRANGE` | 6.165 µs | 26 万/s | 0% |
| `ZRANK` | 5.499 µs | 29 万/s | 0% |
| `ZREM` | 5.499 µs | 24 万/s | 0% |
| `ZREVRANGE` | 9.457 µs | 20 万/s | 0% |
| `ZSCORE` | 723.7 ns | 141 万/s | 0% |
---
## Storage Architecture & Encoding Design
```mermaid
graph TD
Client["Application Code"] --> WeDb["WeDb Instance"]
WeDb --> NS["Namespace Handle<br/>(Zero-Heap Struct)"]
subgraph KeyComposer["Key Composer & Fast Encoding"]
Tag["1-Byte Fast KeyTag (#[repr(u8)])"]
OPPV["OPPV Order-Preserving Varint (1~9B)"]
DefaultBypass["Default NS 0-Prefix Bypass"]
Slot["CRC16 Slot & {hashtag}"]
end
subgraph Engine["Storage Engine Layer"]
Batch["Atomic WriteBatch (Cross-Keyspace WAL)"]
Catalog["Catalog Compact Index"]
end
subgraph Storage["Fjall LSM Dual-Track 4-Partition"]
subgraph DefaultTrack["Default NS Track (db0 / 0-Byte Prefix)"]
DataKS["Data Keyspace<br/>(Default String & Subkeys / 64KB Blocks)"]
MetaKS["Meta Keyspace<br/>(Default Composite Metadata / 4KB Blocks)"]
end
subgraph TenantTrack["Tenant Track (Multi-Tenant & Multi-DB)"]
DataNsKS["Data NS Keyspace<br/>(Tenant String & Subkeys / 64KB Blocks)"]
MetaNsKS["Meta NS Keyspace<br/>(Tenant Metadata & Catalog / 4KB Blocks)"]
end
end
WeDb --> KeyComposer
NS --> KeyComposer
KeyComposer --> Engine
Engine --> DataKS
Engine --> MetaKS
Engine --> DataNsKS
Engine --> MetaNsKS
```
### Dual-Track Storage Partitioning & Zero-Byte Prefix
- **Dual-Track 4-Partition Storage**:
- **Default Track (`data` / `meta`)**: Tailored for single-DB scenarios (`default` / `db 0`), storing keys directly as raw bytes with 0-byte prefix overhead. Composite metadata and subkeys are segregated using 1-Byte `KeyTag`.
- **Tenant Track (`data_ns` / `meta_ns`)**: Dedicated to multi-tenant and multi-DB isolation. Tenant data resides in `data_ns` with encoded tenant prefixes, while tenant metadata and catalog directories reside in `meta_ns`, physically eliminating key collisions.
- **1-Byte Fast Tag (`KeyTag`)**: Composite metadata and subkey prefixes use `#[repr(u8)] KeyTag` encoding (`\x01[key]`), minimizing string prefix overhead.
### Order-Preserving Prefix Varint (OPPV)
- Database indices and tenant numerical IDs use **OPPV (Order-Preserving Prefix Varint)** encoding:
- Values $0 \sim 127$ occupy only 1 byte (significantly smaller than fixed 8-byte big-endian integers).
- Preserves lexicographical byte order equal to numeric value order: $\forall a < b \implies \text{encode}(a) < \text{encode}(b)$, allowing native range scans.
### Numerical Tenant Mapping & Atomic Renaming
- Tenant names map bi-directionally to numerical IDs (`ns_id: u64`) via global metadata.
- **Tenant Renaming (`rename_namespace`)**: Metadata mappings are atomically updated without rewriting underlying data keys.
### Memory-Efficient Streaming Iterators
- **`WeDb::iter(&self) -> Namespaces`**: Iterates distinct tenants via metadata prefixes with $O(1)$ auxiliary memory.
- **`Namespace::iter(&self) -> Dbs`**: Decodes active database indices directly from Catalog directory entries.
---
## Runtime Architecture & Threading Model
The underlying architecture of `wedb_embed` is co-designed specifically for **Thread-per-Core (one thread per CPU core)** asynchronous runtimes based on Linux `io_uring` (such as `compio`), maximizing hardware data locality, lock-free execution, and single-core CPU cache residency.
```mermaid
graph LR
subgraph CompioModel["compio Thread Model (Thread-per-Core)"]
direction TB
C1["CPU Core 0 (Worker 0)<br/>Pinned Core"] --> S1["Local SmallKey Stack Buffer (128B)<br/>L1/L2 Cache Hit (Zero Invalidation)"]
C2["CPU Core 1 (Worker 1)<br/>Pinned Core"] --> S2["Local SmallKey Stack Buffer (128B)<br/>L1/L2 Cache Hit (Zero Invalidation)"]
S1 --> IO1["Direct Synchronous / io_uring<br/>No Work-Stealing ｜ Zero Syscall Overhead"]
S2 --> IO2["Direct Synchronous / io_uring<br/>No Work-Stealing ｜ Zero Syscall Overhead"]
end
subgraph TokioModel["Tokio Work-Stealing Model"]
direction TB
T1["Worker Thread A"] <-->|"Cross-Core Work Stealing<br/>L1/L2 Cache Bouncing ｜ NUMA Migration"| T2["Worker Thread B"]
T1 --> ST["Heap-Allocated State Machine (Send + 'static)<br/>Arc/Mutex Contention ｜ Stack Lifetime Lost"]
T2 --> SB["spawn_blocking Pool Switch<br/>Thread Context Switches ｜ 5~10x Latency Amplification"]
end
```
### Thread-per-Core Architecture Design
- **Stack-Allocated Lifetimes & Zero-Heap Allocation**:<br>
Physical key synthesis utilizes `SmallKey` 128-byte stack buffers and `SubkeyComposer` in-place prefix reuse. In a Thread-per-Core model, execution stays strictly within the current CPU core's stack frame without requesting heap allocations from global memory allocators (such as `jemalloc` or `glibc malloc`), eliminating allocator-level lock contention.
- **CPU Cache Line Locality & Zero Bouncing**:<br>
Worker threads are statically pinned to physical CPU cores without cross-core task migration. Hot data structures (LSM-Tree memtable indices, bloom filter bitsets, and Catalog metadata caches) stay resident in L1/L2 CPU caches, preventing MESI cache coherency protocols from broadcasting invalidation traffic across cores (eliminating Cache Line Bouncing).
- **Lock-Free & Lightweight Metadata Access**:<br>
Namespace and tenant directories are managed via `papaya::HashMap` lock-free concurrent hash maps, ensuring wait-free and lock-free read operations across all threads. Infrequent metadata updates use `parking_lot` adaptive spinlocks that resolve immediately within single-core execution without triggering OS-level futex sleeps or cross-core wakeup overhead.
- **Direct Synchronous Calls Without Cross-Thread Scheduling**:<br>
All engine APIs are direct, synchronous in-memory/disk function calls. In a `compio` single-threaded event loop, microsecond and nanosecond lookups execute directly on the local thread and integrate seamlessly with completion-based `io_uring` I/O, avoiding the runtime overhead of wrapping futures into cross-thread state machines.
### Pitfalls of Multi-Threaded Work-Stealing Runtimes
Employing `wedb_embed` inside general-purpose multi-threaded work-stealing runtimes based on `epoll` (such as `tokio`) introduces distinct physical bottlenecks and resource overhead:
- **Cross-Core Task Stealing & Cache Invalidation**:<br>
Tokio's work-stealing scheduler migrates tasks across worker threads when idle. A single future resumed after an `await` point may run on a different CPU core or across NUMA nodes, causing L1/L2 data and instruction caches to invalidate completely and inducing tail-latency (P99/P999) jitter.
- **Broken Stack Lifetimes & Mandatory Heap Allocations**:<br>
Tokio asynchronous tasks require `Send + 'static` bounds. Stack-allocated borrowed references (such as `&[u8]` slices and stack `SmallKey` buffers) cannot cross `await` boundaries without being promoted to heap memory (`Box`, `Arc`, or `Vec<u8>`), negating the zero-heap allocation advantages of embedded execution.
- **Event Loop Starvation & Blocking Thread Pool Overhead**:<br>
In an `epoll`-based runtime, executing synchronous disk I/O or CPU-bound index lookups directly inside worker threads blocks the event loop, starving concurrent network connections. Offloading operations via `tokio::task::spawn_blocking` introduces thread context switches, cross-thread channel (IPC) transfers, and task reschedulings, amplifying microsecond operations by 5~10x.
- **Multi-Core Bus Locking & Mutex Contention**:<br>
When multiple threads concurrently contend for shared instances, frequent atomic operations (CAS) and mutex locking across cores cause memory bus contention and lock storms, reducing peak concurrent throughput.
---
## Multi-Tenant & Multi-DB Isolation
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/multitenant_demo", [])?;
// 1. Tenant-isolated handles (zero-heap structs)
let ns_apple = db.namespace("tenant_apple");
let ns_google = db.namespace("tenant_google");
ns_apple.set(b"config:theme", b"dark", &[])?;
ns_google.set(b"config:theme", b"light", &[])?;
// 2. Physically isolated lookups
assert_eq!(ns_apple.get(b"config:theme")?.unwrap(), b"dark");
assert_eq!(ns_google.get(b"config:theme")?.unwrap(), b"light");
// 3. Multi-DB selection (e.g. SELECT 1)
let db1 = db.select_db(1)?;
db1.set(b"db1_key", b"value1", &[])?;
// 4. Iterate active databases under tenant (streaming iterator)
for db_idx in &ns_apple {
println!("Active DB: {}", db_idx);
}
// 5. Atomic tenant renaming (zero data rewrite)
db.rename_namespace("tenant_apple", "tenant_apple_v2")?;
// 6. Keyspace management
let count = ns_apple.key_count()?;
let keys = ns_apple.keys("config:*")?;
let exists = ns_apple.exists(&[b"config:theme"])?;
ns_apple.del(&[b"config:theme"])?;
ns_apple.clear()?; // Cascade clear all tenant data
Ok(())
}
```
---
## Data Structures & Operations
### Initialization & Configuration
```rust
use wedb_embed::{Conf, PersistMode, Result, WeDb};
fn main() -> Result<()> {
// Open with custom configuration enum list (LZ4 compression enabled by default)
let db = WeDb::open(
"./data/db_demo",
[
Conf::CacheSize(128 * 1024 * 1024), // 128MB LSM block cache
Conf::ManualJournalPersist(false), // Auto-flush WAL in background
Conf::WorkerThreads(2), // Background worker threads
],
)?;
// Persistence and active expiration tasks
db.persist(PersistMode::SyncAll)?; // Synchronously persist WAL and dirty pages
db.active_expire_cycle(100)?; // Trigger active key expiration sampling
Ok(())
}
```
### Key-Value & Strings
```rust
use wedb_embed::{prelude::*, string::Set, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/kv_demo", [])?;
// Basic read, write, and TTL expiration
db.set(b"site", b"webc.site", &[])?;
let val = db.get(b"site")?;
db.setex(b"temp_token", b"xyz", 3600_000)?; // Millisecond TTL
// Conditional write (NX / XX)
db.set(b"lock", b"1", &[Set::Nx])?;
// Numerical increment & decrement
db.incr(b"counter")?;
db.incrby(b"counter", 10)?;
db.decrby(b"counter", 5)?;
// Batch & atomic operations
db.mset(&[(b"k1", b"v1"), (b"k2", b"v2")])?;
let values = db.mget(&[b"k1", b"k2"])?;
let len = db.strlen(b"site")?;
let old_val = db.getset(b"site", b"new_site")?;
let del_val = db.getdel(b"site")?;
db.cas(b"site", b"new_site", b"final_site", 0)?; // Compare and Swap
Ok(())
}
```
### Hash Map & Field-Level TTL
```rust
use wedb_embed::{hash::HExpire, prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/hash_demo", [])?;
// Field write & read
db.hset(b"user:100", &[(b"name", b"Alice"), (b"age", b"20")])?;
let name = db.hget(b"user:100", b"name")?;
// Batch field operations
db.hmset(b"user:100", &[(b"city", b"Beijing"), (b"role", b"Admin")])?;
let fields = db.hmget(b"user:100", &[b"name", b"city"])?;
// Counter increment and introspection
db.hincrby(b"user:100", b"age", 1)?;
let exists = db.hexists(b"user:100", b"name")?;
let len = db.hlen(b"user:100")?;
let all = db.hgetall(b"user:100")?;
let keys = db.hkeys(b"user:100")?;
let vals = db.hvals(b"user:100")?;
db.hdel(b"user:100", &[b"role"])?;
// Field-level independent TTL
db.hexpire(b"user:100", &[b"name"], 3600, HExpire::None)?;
let ttls = db.httl(b"user:100", &[b"name"])?;
Ok(())
}
```
### List
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/list_demo", [])?;
// Push and pop operations
db.lpush(b"tasks", &[b"task1", b"task2"])?;
db.rpush(b"tasks", &[b"task3"])?;
let first = db.lpop_one(b"tasks")?;
let last = db.rpop_one(b"tasks")?;
// Range, trim, and set
let len = db.llen(b"tasks")?;
let items = db.lrange(b"tasks", 0, -1)?;
db.ltrim(b"tasks", 0, 10)?;
db.lset(b"tasks", 0, b"updated_task")?;
Ok(())
}
```
### Set
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/set_demo", [])?;
db.sadd(b"tags", &[b"rust", b"database", b"lsm"])?;
let is_member = db.sismember(b"tags", b"rust")?;
let count = db.scard(b"tags")?;
let members = db.smembers(b"tags")?;
db.srem(b"tags", &[b"lsm"])?;
let popped = db.spop(b"tags", 1)?;
// Set algebra operations
db.sadd(b"tags_other", &[b"rust", b"storage"])?;
let inter = db.sinter(&[b"tags", b"tags_other"])?;
let union = db.sunion(&[b"tags", b"tags_other"])?;
let diff = db.sdiff(&[b"tags", b"tags_other"])?;
Ok(())
}
```
### Sorted Set
```rust
use wedb_embed::{prelude::*, zset::RangeScoreSpec, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/zset_demo", [])?;
// Add members and scores
db.zadd(b"leaderboard", &[(100.0, b"player1"), (200.0, b"player2")], &[])?;
db.zincrby(b"leaderboard", 50.0, b"player1")?;
// Score and rank queries
let score = db.zscore(b"leaderboard", b"player1")?;
let rank = db.zrank(b"leaderboard", b"player1")?;
let rev_rank = db.zrevrank(b"leaderboard", b"player1")?;
// Range queries and count
let top = db.zrange(b"leaderboard", 0, 10)?;
let count = db.zcount(b"leaderboard", &RangeScoreSpec::new(100.0, 300.0))?;
let card = db.zcard(b"leaderboard")?;
db.zrem(b"leaderboard", &[b"player1"])?;
Ok(())
}
```
### Bitmap & Bitfield
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/bitmap_demo", [])?;
// Bit manipulation
db.setbit(b"online_users", 1001, 1)?;
let is_online = db.getbit(b"online_users", 1001)?;
let online_count = db.bitcount(b"online_users", None, None)?;
let first_online = db.bitpos(b"online_users", 1, None, None)?;
Ok(())
}
```
### JSON & JSONPath
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/json_demo", [])?;
// JSON write and JSONPath queries
db.json_set(b"doc:1", "$", r#"{"user":{"name":"Alice","age":25,"roles":["admin"]}}"#)?;
let name = db.json_get(b"doc:1", Some("$.user.name"))?;
db.json_numincrby(b"doc:1", "$.user.age", "1.0")?;
db.json_arrappend(b"doc:1", "$.user.roles", &[r#""editor""#])?;
Ok(())
}
```
### Bloom & Cuckoo Filters
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/filter_demo", [])?;
// Bloom Filter
db.bf_reserve(b"bf_filter", 0.01, 10000, 2)?;
db.bf_add(b"bf_filter", b"item_1")?;
let exists = db.bf_exists(b"bf_filter", b"item_1")?;
// Cuckoo Filter (supports dynamic deletion)
db.cf_reserve(b"cf_filter", 10000, 2, 500, 1)?;
db.cf_add(b"cf_filter", b"item_2")?;
let cf_exists = db.cf_exists(b"cf_filter", b"item_2")?;
db.cf_del(b"cf_filter", b"item_2")?;
Ok(())
}
```
### TimeSeries & Aggregations
```rust
use wedb_embed::{
prelude::*,
timeseries::{AggregationType, DuplicatePolicy, TSRangeOption},
Result, WeDb,
};
fn main() -> Result<()> {
let db = WeDb::open("./data/timeseries_demo", [])?;
// Create series
db.ts_create(b"cpu:usage", 0, DuplicatePolicy::Last, &[("host", "server-1")])?;
db.ts_add(b"cpu:usage", 1700000000000, 42.5)?;
let latest = db.ts_get(b"cpu:usage")?;
// Window downsampling query (Gorilla delta-of-delta compression)
let samples = db.ts_range_opt(
b"cpu:usage",
&TSRangeOption {
from_timestamp: 0,
to_timestamp: 1800000000000,
aggregation: Some((AggregationType::Avg, 60000)),
..Default::default()
},
)?;
Ok(())
}
```
### Geospatial
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/geo_demo", [])?;
// Coordinate storage and distance calculation
db.geoadd(
b"cities",
&[
(116.4074, 39.9042, b"Beijing"),
(121.4737, 31.2304, b"Shanghai"),
],
)?;
let dist_km = db.geodist(b"cities", b"Beijing", b"Shanghai", Some("km"))?;
let hash = db.geohash(b"cities", &[b"Beijing"])?;
Ok(())
}
```
### HyperLogLog
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/hll_demo", [])?;
db.pfadd(b"hll_uv", &[b"user_1", b"user_2", b"user_3"])?;
let uv_count = db.pfcount(&[b"hll_uv"])?;
db.pfmerge(b"hll_merged", &[b"hll_uv"])?;
Ok(())
}
```
### T-Digest
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/tdigest_demo", [])?;
db.tdigest_create(b"latencies", 100.0)?;
db.tdigest_add(b"latencies", &[12.5, 15.0, 18.2, 45.0, 99.9])?;
let p95 = db.tdigest_quantile(b"latencies", &[0.95])?;
let cdf = db.tdigest_cdf(b"latencies", &[20.0])?;
Ok(())
}
```
### SortedInt
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/sortedint_demo", [])?;
// 64-bit compact integer set
db.si_add(b"post:100:likes", &[1001, 1002, 1003, 1004])?;
let has_liked = db.si_exists(b"post:100:likes", 1001)?;
let count = db.si_card(b"post:100:likes")?;
let top = db.si_range(b"post:100:likes", 0, 0, 10, false)?;
Ok(())
}
```
### Streams & Consumer Groups
```rust
use wedb_embed::{prelude::*, stream::StreamId, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/stream_demo", [])?;
// Append entry
let entry_id = db.xadd_simple(
b"event_stream",
None,
&[(b"action", b"login"), (b"uid", b"1001")],
)?;
let len = db.xlen(b"event_stream")?;
let entries = db.xrange(b"event_stream", StreamId::MIN, StreamId::MAX, Some(10))?;
// Consumer groups & pending entries
db.xgroup_create(b"event_stream", "group_workers", "0-0", false, None)?;
let pending = db.xpending_summary(b"event_stream", "group_workers")?;
Ok(())
}
```
### Full-Text Search & Vector Retrieval
```rust
use wedb_embed::{
prelude::*,
search::{
DistanceMetric, FtCreate, FtSearch, IndexField, IndexFieldType, IndexOnDataType,
SearchIndexManager, VectorAlgorithm, VectorFieldMetadata, VectorType,
},
Result, WeDb,
};
fn main() -> Result<()> {
let db = WeDb::open("./data/search_demo", [])?;
let mut search_mgr = SearchIndexManager::new();
// 1. Create multi-field schema (Inverted Text + HNSW Vector)
let mut schema = FtCreate::new("idx_articles", IndexOnDataType::Json);
schema.prefixes.push("article:".to_string());
schema.fields.push(IndexField {
name: "title".to_string(),
alias: None,
field_type: IndexFieldType::Text,
weight: 1.0,
sortable: false,
noindex: false,
separator: None,
casesensitive: false,
withsuffixtrie: false,
unf: false,
vector_meta: None,
});
schema.fields.push(IndexField {
name: "vec".to_string(),
alias: None,
field_type: IndexFieldType::Vector,
weight: 1.0,
sortable: false,
noindex: false,
separator: None,
casesensitive: false,
withsuffixtrie: false,
unf: false,
vector_meta: Some(VectorFieldMetadata {
algorithm: VectorAlgorithm::Hnsw,
vector_type: VectorType::Float32,
dim: 4,
distance_metric: DistanceMetric::Cosine,
m: 16,
ef_construction: 200,
ef_runtime: 10,
block_size: 1024,
initial_cap: 1000,
}),
});
search_mgr.create_index_from_opts(schema)?;
// 2. Execute text and vector queries
let res = search_mgr.search("idx_articles", "@title:database", &FtSearch::default())?;
println!("Found matching docs: {}", res.total);
Ok(())
}
```
---
## Tech Stack
- **Language**: Rust Edition 2024
- **Storage Engine**: `fjall` (LSM-Tree persistence engine)
- **JSON Engine**: `sonic-rs` (SIMD-accelerated parser)
- **Non-Cryptographic Hash**: `rapidhash`
- **Serialization**: `bitcode` (compact binary encoding)
- **Strings & Memory**: `hipstr` (compact string and zero-copy borrowing)
- **Concurrent Hash Map**: `papaya` (lock-free concurrent dictionary)
- **Bitwise & Collections**: `roaring`, `memchr`, `crc32fast`, `fastrand`
- **Timestamps**: `coarsetime`, `ts_`
- **Number Formatter**: `zmij`, `itoa`
- **Enum Derives**: `strum`
- **Error Management**: `thiserror`
---
<a id="zh"></a>
<h1 id="wedb_embed">wedb_embed</h1>
嵌入式数据库引擎，提供 Redis 兼容数据结构与接口，基于 [fjall](https://github.com/fjall-rs/fjall) LSM-Tree 存储引擎构建。
<p align="center">
<img src="https://fastly.jsdelivr.net/gh/webc-fs/-@13/bhQF-zCHwUFzGJ-KgEXg.svg" alt="wedb_embed vs Redis 性能与资源对比" width="100%">
<br>
<sub><b>测试环境</b>: CPU: Apple M2 Max (12核) ｜ 内存: 64.0 GB ｜ 系统: macOS 26.5.1 (Darwin 25.5.0) ｜ Rust: 1.98.0 (88d9e12ae 2026-08-18) ｜ Redis: v8.10.1</sub>
</p>
---
## 为什么需要嵌入式 Redis 引擎
- [为什么需要嵌入式 Redis 引擎](#为什么需要嵌入式-redis-引擎)
- [快速上手](#快速上手)
- [添加依赖](#添加依赖)
- [基础读写示例](#基础读写示例)
- [性能与资源实测对比](#性能与资源实测对比)
- [macOS (Apple M2 Max)](#macos-apple-m2-max)
- [硬件与测试环境](#硬件与测试环境)
- [真实物理落盘与内存占用实测 (5GB 数据规模)](#真实物理落盘与内存占用实测-5gb-数据规模)
- [wedb_embed vs Redis 核心指令性能对比](#wedb_embed-vs-redis-核心指令性能对比)
- [wedb_embed 性能回归测试](#wedb_embed-性能回归测试)
- [存储架构与编码设计](#存储架构与编码设计)
- [双轨存储分区与零前缀](#双轨存储分区与零前缀)
- [保序变长整型编码](#保序变长整型编码)
- [租户数字映射与原子重命名](#租户数字映射与原子重命名)
- [内存友好流式迭代](#内存友好流式迭代)
- [运行时生态与线程模型设计](#运行时生态与线程模型设计)
- [一线程一核心架构设计](#一线程一核心架构设计)
- [传统多线程工作窃取运行时问题分析](#传统多线程工作窃取运行时问题分析)
- [多租户与分库隔离](#多租户与分库隔离)
- [数据结构与接口演示](#数据结构与接口演示)
- [初始化与配置](#初始化与配置)
- [键值与字符串](#键值与字符串)
- [哈希与字段级过期](#哈希与字段级过期)
- [列表](#列表)
- [集合](#集合)
- [有序集合](#有序集合)
- [位图与位域](#位图与位域)
- [JSON 与路径查询](#json-与路径查询)
- [布隆与布谷鸟过滤器](#布隆与布谷鸟过滤器)
- [时序数据与聚合](#时序数据与聚合)
- [地理空间位置](#地理空间位置)
- [基数统计](#基数统计)
- [分位数统计](#分位数统计)
- [紧凑整型集合](#紧凑整型集合)
- [消息流与消费组](#消息流与消费组)
- [全文检索与向量检索](#全文检索与向量检索)
- [技术堆栈](#技术堆栈)
在后端服务、CLI 工具、边缘计算与桌面端应用中，开发者经常需要使用丰富的数据结构（例如哈希表、有序集合排行榜、消息队列、位图与时序序列）。传统的方案是部署独立的 Redis 进程，通过网络或本地套接字进行通信。这种架构在单机环境下存在以下物理瓶颈：
- **进程间通信与协议开销**：每次读写都需要经过数据序列化、操作系统套接字缓冲区、进程上下文切换、RESP 协议解析与事件循环处理。即使在本地主机上，套接字往返延迟通常也在 20~50 微秒区间，并消耗额外的 CPU 周期。
- **物理内存成本与容量约束**：Redis 将全量数据与内部指针结构常驻于物理内存中。当数据规模增长到数十 GB 时，内存硬件成本高昂，且受限于单机物理 RAM 容量。开启 AOF 或 RDB 持久化时，后台保存机制还会引发写时复制的额外内存开销。
- **部署与运维复杂度**：独立进程需要额外的进程守护、端口监听、配置分发与健康检查逻辑，增加了软件交付与部署的维护成本。
`wedb_embed` 将存储引擎直接集成到应用程序的进程空间中，改变了数据存储与访问路径：
- **进程内直接调用**：所有数据操作直接通过 Rust 函数调用完成，消除套接字通信、系统调用与跨进程上下文切换。在同等硬件下，核心指令的 P95 延迟从 Redis 的数十微秒降低至纳秒到微秒级。
- **LSM-Tree 磁盘持久化与冷热分层**：基于 LSM-Tree 存储引擎与 LZ4 分块压缩，活跃数据保留在内存缓存中，冷数据与全量数据经过压缩持久化落盘。在 2GB 真实数据实测中，常驻内存 RSS 占用由 Redis 的 1951 MB 降低至 234 MB（减少 88%），物理落盘体积节省 38%。
- **全量 Redis 数据结构支持**：在底层键值引擎之上，实现了 16 种复合数据模型（包含 String、Hash 字段级 TTL、List、Set、ZSet、Bitmap、JSON、Bloom/Cuckoo 过滤器、TimeSeries、Geo、HyperLogLog、TDigest、SortedInt、Stream、全文检索与 HNSW 向量检索）。
- **多租户与多库物理隔离**：原生支持 $2^{64}$ 个独立租户与分库，租户重命名仅需原子修改元数据映射（$O(1)$ 时间复杂度），无需重写底层数据。
- **数据一致性保证**：基于预写日志 WAL 与跨分区原子批处理 WriteBatch，保障断电与崩溃场景下的数据完整性。
---
## 快速上手
### 添加依赖
```bash
cargo add wedb_embed
```
### 基础读写示例
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
// 打开数据库
let db = WeDb::open("./data/quickstart_db", [])?;
// 字符串读写
db.set(b"site", b"webc.site", &[])?;
let val = db.get(b"site")?;
assert_eq!(val.as_deref(), Some(&b"webc.site"[..]));
// 哈希结构操作
db.hset(b"user:1", &[(b"name", b"Alice"), (b"age", b"20")])?;
let age = db.hget(b"user:1", b"age")?;
assert_eq!(age.as_deref(), Some(&b"20"[..]));
// 有序集合与排行榜
db.zadd(b"rank", &[(100.0, b"player1"), (200.0, b"player2")], &[])?;
let top = db.zrange(b"rank", 0, 10)?;
assert_eq!(top.len(), 2);
Ok(())
}
```
[点此查看更多演示](../examples)
---
- [为什么需要嵌入式 Redis 引擎](#为什么需要嵌入式-redis-引擎)
- [快速上手](#快速上手)
- [添加依赖](#添加依赖)
- [基础读写示例](#基础读写示例)
- [性能与资源实测对比](#性能与资源实测对比)
- [存储架构与编码设计](#存储架构与编码设计)
- [双轨存储分区与零前缀](#双轨存储分区与零前缀)
- [保序变长整型编码](#保序变长整型编码)
- [租户数字映射与原子重命名](#租户数字映射与原子重命名)
- [内存友好流式迭代](#内存友好流式迭代)
- [运行时生态与线程模型设计](#运行时生态与线程模型设计)
- [一线程一核心架构设计](#一线程一核心架构设计)
- [传统多线程工作窃取运行时问题分析](#传统多线程工作窃取运行时问题分析)
- [多租户与分库隔离](#多租户与分库隔离)
- [数据结构与接口演示](#数据结构与接口演示)
- [初始化与配置](#初始化与配置)
- [键值与字符串](#键值与字符串)
- [哈希与字段级过期](#哈希与字段级过期)
- [列表](#列表)
- [集合](#集合)
- [有序集合](#有序集合)
- [位图与位域](#位图与位域)
- [JSON 与路径查询](#json-与路径查询)
- [布隆与布谷鸟过滤器](#布隆与布谷鸟过滤器)
- [时序数据与聚合](#时序数据与聚合)
- [地理空间位置](#地理空间位置)
- [基数统计](#基数统计)
- [分位数统计](#分位数统计)
- [紧凑整型集合](#紧凑整型集合)
- [消息流与消费组](#消息流与消费组)
- [全文检索与向量检索](#全文检索与向量检索)
- [技术堆栈](#技术堆栈)
---
## 性能与资源实测对比
### macOS (Apple M2 Max)
#### 硬件与测试环境
CPU: Apple M2 Max (12核)<br>
内存: 64.0 GB<br>
系统: macOS 26.5.1 (Darwin 25.5.0)<br>
Rust: 1.98.0 (88d9e12ae 2026-08-18)<br>
Redis: v8.10.1
#### 真实物理落盘与内存占用实测 (5GB 数据规模)
| **测试数据规模** | 5,000,000 条全格式结构化数据 | 5,000,000 条全格式结构化数据 | 14 种数据格式等比实测 |
| **原始数据载荷** | 4377 MB | 4377 MB | 真实结构化载荷 |
| **实际物理落盘大小** | **4791 MB** | **7791 MB** | **节省 39%** |
| **进程常驻内存 (RSS)** | **508 MB** | **4918 MB** | **节省 90%** |
#### wedb_embed vs Redis 核心指令性能对比
| `SET` | 9.1 us | 47.3 us | **5.2x** |
| `GET` | 0.83 us | 41.8 us | **50.2x** |
| `MSET` | 52.2 us | 54.6 us | **1.0x** |
| `MGET` | 2.4 us | 43.3 us | **17.9x** |
| `INCRBY` | 0.58 us | 48.1 us | **82.8x** |
| `DECRBY` | 0.58 us | 45.9 us | **79.8x** |
| `APPEND` | 0.78 us | 49.4 us | **63.4x** |
| `STRLEN` | 0.24 us | 40.9 us | **168.1x** |
| `GETDEL` | 10.1 us | 94.5 us | **9.4x** |
| `GETRANGE` | 0.25 us | 42.8 us | **171.3x** |
| `SETRANGE` | 0.59 us | 45.0 us | **75.9x** |
| `HSET` | 3.0 us | 48.4 us | **16.0x** |
| `HGET` | 0.70 us | 47.0 us | **67.2x** |
| `HMGET` | 2.7 us | 45.3 us | **17.0x** |
| `HEXISTS` | 0.64 us | 45.7 us | **71.7x** |
| `HLEN` | 0.47 us | 43.0 us | **90.7x** |
| `HDEL` | 4.9 us | 43.0 us | **8.7x** |
| `HGETALL` | 3.2 us | 44.1 us | **13.9x** |
| `HKEYS` | 3.1 us | 45.8 us | **14.7x** |
| `HVALS` | 3.1 us | 42.8 us | **13.7x** |
| `HINCRBY` | 3.0 us | 45.6 us | **15.3x** |
| `LPUSH` | 2.9 us | 44.1 us | **15.0x** |
| `RPUSH` | 3.4 us | 43.8 us | **13.0x** |
| `LPOP` | 3.0 us | 44.7 us | **14.7x** |
| `RPOP` | 3.2 us | 43.5 us | **13.8x** |
| `LLEN` | 0.46 us | 40.8 us | **89.1x** |
| `LRANGE` | 2.6 us | 42.4 us | **16.2x** |
| `LINDEX` | 0.67 us | 41.0 us | **61.3x** |
| `LSET` | 2.2 us | 44.0 us | **19.9x** |
| `LREM` | 14.0 us | 93.9 us | **6.7x** |
| `LTRIM` | 2.8 us | 43.1 us | **15.4x** |
| `SADD` | 2.3 us | 44.9 us | **19.2x** |
| `SREM` | 4.8 us | 46.5 us | **9.7x** |
| `SISMEMBER` | 0.65 us | 41.3 us | **63.9x** |
| `SCARD` | 0.48 us | 41.1 us | **86.3x** |
| `SMEMBERS` | 3.2 us | 41.8 us | **13.0x** |
| `SPOP` | 9.2 us | 88.4 us | **9.6x** |
| `SRANDMEMBER` | 3.3 us | 40.9 us | **12.4x** |
| `ZADD` | 3.7 us | 43.9 us | **11.8x** |
| `ZSCORE` | 0.72 us | 44.3 us | **61.3x** |
| `ZRANGE` | 3.7 us | 45.8 us | **12.2x** |
| `ZCARD` | 0.49 us | 40.8 us | **82.5x** |
| `ZCOUNT` | 3.2 us | 41.6 us | **13.1x** |
| `ZINCRBY` | 3.9 us | 44.6 us | **11.4x** |
| `ZRANK` | 3.6 us | 41.7 us | **11.7x** |
| `ZREVRANGE` | 5.3 us | 46.0 us | **8.7x** |
| `ZPOPMIN` | 9.0 us | 97.0 us | **10.8x** |
| `ZREM` | 5.1 us | 43.0 us | **8.4x** |
| `SETBIT` | 9.5 us | 55.4 us | **5.8x** |
| `GETBIT` | 0.42 us | 47.5 us | **112.7x** |
| `BITCOUNT` | 0.44 us | 46.0 us | **104.3x** |
| `BITPOS` | 0.45 us | 50.0 us | **111.0x** |
| `PFADD` | 4.5 us | 44.9 us | **10.0x** |
| `PFCOUNT` | 32.3 us | 41.5 us | **1.3x** |
| `GEOADD` | 4.0 us | 46.3 us | **11.6x** |
| `GEODIST` | 0.86 us | 41.7 us | **48.6x** |
| `GEOPOS` | 0.65 us | 41.3 us | **63.2x** |
| `GEOHASH` | 0.70 us | 41.1 us | **58.9x** |
| `XADD` | 2.7 us | 44.9 us | **16.8x** |
| `XLEN` | 0.45 us | 40.7 us | **90.3x** |
| `XRANGE` | 3.8 us | 47.8 us | **12.5x** |
| `XREAD` | 3.7 us | 49.1 us | **13.1x** |
| `XDEL` | 5.3 us | 88.3 us | **16.6x** |
| `DEL` | 4.1 us | 35.6 us | **8.6x** |
| `EXISTS` | 0.20 us | 35.1 us | **172.9x** |
| `EXPIRE` | 2.0 us | 42.0 us | **21.0x** |
| `TTL` | 0.25 us | 42.0 us | **168.1x** |
| `JSON.SET` | 4.1 us | 45.8 us | **11.1x** |
| `JSON.GET` | 1.3 us | 46.4 us | **36.6x** |
| `JSON.DEL` | 11.3 us | 101.5 us | **9.0x** |
| `JSON.NUMINCRBY` | 4.2 us | 48.2 us | **11.4x** |
| `JSON.ARRLEN` | 1.1 us | 46.4 us | **42.1x** |
| `JSON.TYPE` | 1.1 us | 42.5 us | **37.8x** |
| `BF.ADD` | 20.6 us | 48.2 us | **2.3x** |
| `BF.EXISTS` | 0.63 us | 37.4 us | **59.7x** |
| `BF.INFO` | 0.44 us | 48.6 us | **110.8x** |
| `CF.ADD` | 4.1 us | 39.7 us | **9.7x** |
| `CF.EXISTS` | 0.66 us | 35.5 us | **53.6x** |
| `CF.DEL` | 7.6 us | 84.7 us | **11.1x** |
| `TDIGEST.ADD` | 3.3 us | 41.2 us | **12.5x** |
| `TDIGEST.QUANTILE` | 0.83 us | 41.4 us | **49.7x** |
| `TDIGEST.BYRANK` | 0.83 us | 43.4 us | **52.1x** |
| `TDIGEST.CDF` | 1.0 us | 45.4 us | **43.6x** |
| `TS.ADD` | 6.5 us | 44.8 us | **6.9x** |
| `TS.GET` | 2.2 us | 43.4 us | **19.6x** |
| `TS.RANGE` | 10.6 us | 70.6 us | **6.7x** |
| `TS.INCRBY` | 9.8 us | 45.2 us | **4.6x** |
| `FT.SEARCH` | 19.8 us | 77.2 us | **3.9x** |
| `FT.TAG` | 19.6 us | 66.4 us | **3.4x** |
| `VECTOR.KNN` | 2.2 us | 64.8 us | **28.8x** |
#### wedb_embed 性能回归测试
| `BITCOUNT` | 437.2 ns | 237 万/s | 0% |
| `BITPOS` | 1.135 µs | 96 万/s | 0% |
| `GETBIT` | 468.4 ns | 218 万/s | 0% |
| `SETBIT` | 117.7 µs | 9 万/s | 0% |
| `BF.ADD` | 50.79 µs | 3 万/s | 0% |
| `BF.EXISTS` | 713.2 ns | 150 万/s | 0% |
| `BF.INFO` | 432 ns | 233 万/s | 0% |
| `CF.ADD` | 55.83 µs | 16 万/s | 0% |
| `CF.DEL` | 34.16 µs | 9 万/s | 0% |
| `CF.EXISTS` | 812.2 ns | 138 万/s | 0% |
| `BATCH_COMMIT` | 34.66 µs | 16 万/s | 0% |
| `DEL` | 157.8 µs | 5 万/s | 0% |
| `EXISTS` | 170.2 ns | 601 万/s | 0% |
| `EXPIRE` | 6.041 µs | 29 万/s | 0% |
| `NAMESPACE` | 11.45 µs | 19 万/s | 0% |
| `TTL` | 227.5 ns | 450 万/s | 0% |
| `GEOADD` | 227.4 µs | 16 万/s | 0% |
| `GEODIST` | 1.244 µs | 84 万/s | 0% |
| `GEOHASH` | 728.9 ns | 139 万/s | 0% |
| `GEOPOS` | 984 ns | 123 万/s | 0% |
| `HDEL` | 5.208 µs | 26 万/s | 0% |
| `HEXISTS` | 640.3 ns | 159 万/s | 0% |
| `HGET` | 728.8 ns | 142 万/s | 0% |
| `HGETALL` | 30.7 µs | 27 万/s | 0% |
| `HINCRBY` | 172.7 µs | 19 万/s | 0% |
| `HKEYS` | 33.04 µs | 28 万/s | 0% |
| `HLEN` | 471 ns | 216 万/s | 0% |
| `HMGET` | 2.728 µs | 40 万/s | 0% |
| `HSET` | 176.5 µs | 19 万/s | 0% |
| `HVALS` | 25.29 µs | 30 万/s | 0% |
| `PFADD` | 191.3 µs | 6 万/s | 0% |
| `PFCOUNT` | 41.04 µs | 3 万/s | 0% |
| `PFMERGE` | 40.95 µs | 7 万/s | 0% |
| `JSON.ARRLEN` | 43.45 µs | 71 万/s | 0% |
| `JSON.DEL` | 12.83 µs | 11 万/s | 0% |
| `JSON.GET` | 26.49 µs | 63 万/s | 0% |
| `JSON.NUMINCRBY` | 48.29 µs | 18 万/s | 0% |
| `JSON.SET` | 64.58 µs | 13 万/s | 0% |
| `JSON.TYPE` | 1.676 µs | 87 万/s | 0% |
| `LINDEX` | 833 ns | 135 万/s | 0% |
| `LLEN` | 484 ns | 211 万/s | 0% |
| `LPOP` | 21.41 µs | 6 万/s | 0% |
| `LPUSH` | 191.9 µs | 20 万/s | 0% |
| `LRANGE` | 4.582 µs | 32 万/s | 0% |
| `LREM` | 224 µs | 5 万/s | 0% |
| `LSET` | 4.083 µs | 32 万/s | 0% |
| `LTRIM` | 7.29 µs | 33 万/s | 0% |
| `RPOP` | 8.416 µs | 13 万/s | 0% |
| `RPUSH` | 132.6 µs | 19 万/s | 0% |
| `FT.SEARCH` | 42.2 µs | 4 万/s | 0% |
| `FT.TAG` | 27.66 µs | 4 万/s | 0% |
| `SADD` | 150.6 µs | 16 万/s | 0% |
| `SCARD` | 520.4 ns | 202 万/s | 0% |
| `SISMEMBER` | 765.3 ns | 138 万/s | 0% |
| `SMEMBERS` | 6.291 µs | 30 万/s | 0% |
| `SPOP` | 17.95 µs | 10 万/s | 0% |
| `SRANDMEMBER` | 6.249 µs | 22 万/s | 0% |
| `SREM` | 4.916 µs | 25 万/s | 0% |
| `SI.ADD` | 187 µs | 19 万/s | 0% |
| `SI.CARD` | 585.6 ns | 210 万/s | 0% |
| `SI.EXISTS` | 822.5 ns | 123 万/s | 0% |
| `SI.RANGE` | 17.49 µs | 15 万/s | 0% |
| `SI.REM` | 158.2 µs | 6 万/s | 0% |
| `APPEND` | 609 ns | 175 万/s | 0% |
| `DECRBY` | 666.4 ns | 166 万/s | 0% |
| `GET` | 260.1 ns | 418 万/s | 0% |
| `GETDEL` | 180.7 µs | 7 万/s | 0% |
| `GETRANGE` | 372.1 ns | 500 万/s | 0% |
| `INCRBY` | 775.7 ns | 147 万/s | 0% |
| `MGET` | 2.666 µs | 42 万/s | 0% |
| `MSET` | 172.2 µs | 2 万/s | 0% |
| `SET` | 121 µs | 8 万/s | 0% |
| `SETRANGE` | 872.1 ns | 195 万/s | 0% |
| `STRLEN` | 197.6 ns | 516 万/s | 0% |
| `XADD` | 176.1 µs | 21 万/s | 0% |
| `XDEL` | 169.2 µs | 10 万/s | 0% |
| `XLEN` | 486.7 ns | 207 万/s | 0% |
| `XRANGE` | 12.33 µs | 11 万/s | 0% |
| `XREAD` | 17.29 µs | 11 万/s | 0% |
| `TDIGEST.ADD` | 9.791 µs | 19 万/s | 0% |
| `TDIGEST.BYRANK` | 13.83 µs | 89 万/s | 0% |
| `TDIGEST.CDF` | 10.91 µs | 80 万/s | 0% |
| `TDIGEST.QUANTILE` | 12.49 µs | 83 万/s | 0% |
| `TS.ADD` | 66.62 µs | 12 万/s | 0% |
| `TS.GET` | 5.291 µs | 43 万/s | 0% |
| `TS.INCRBY` | 53.62 µs | 8 万/s | 0% |
| `TS.RANGE` | 53.87 µs | 3 万/s | 0% |
| `VECTOR.KNN` | 8.165 µs | 34 万/s | 0% |
| `ZADD` | 169.9 µs | 18 万/s | 0% |
| `ZCARD` | 538.7 ns | 226 万/s | 0% |
| `ZCOUNT` | 5.874 µs | 31 万/s | 0% |
| `ZINCRBY` | 146.8 µs | 19 万/s | 0% |
| `ZPOPMIN` | 14.79 µs | 13 万/s | 0% |
| `ZRANGE` | 6.165 µs | 26 万/s | 0% |
| `ZRANK` | 5.499 µs | 29 万/s | 0% |
| `ZREM` | 5.499 µs | 24 万/s | 0% |
| `ZREVRANGE` | 9.457 µs | 20 万/s | 0% |
| `ZSCORE` | 723.7 ns | 141 万/s | 0% |
---
## 存储架构与编码设计
```mermaid
graph TD
Client["应用业务代码"] --> WeDb["WeDb 数据库实例"]
WeDb --> NS["Namespace 句柄<br/>(零堆分配结构体)"]
subgraph KeyComposer["键编排与紧凑编码"]
Tag["1 字节键标签 (#[repr(u8)])"]
OPPV["保序变长整型 (1~9 字节)"]
DefaultBypass["默认库 0 字节前缀直通"]
Slot["哈希槽与哈希标签"]
end
subgraph Engine["存储引擎与事务层"]
Batch["原子写批处理 (跨分区 WAL)"]
Catalog["紧凑元数据目录"]
end
subgraph Storage["Fjall LSM 双轨 4 分区存储"]
subgraph DefaultTrack["默认库轨道 (db 0 / 0 字节物理前缀)"]
DataKS["数据分区<br/>(字符串数据与复合子键 / 64KB 块)"]
MetaKS["元数据分区<br/>(复合结构元数据 / 4KB 块)"]
end
subgraph TenantTrack["租户轨道 (多租户与多 DB 隔离)"]
DataNsKS["租户数据分区<br/>(租户字符串与子键 / 64KB 块)"]
MetaNsKS["租户元数据分区<br/>(租户元数据与目录 / 4KB 块)"]
end
end
WeDb --> KeyComposer
NS --> KeyComposer
KeyComposer --> Engine
Engine --> DataKS
Engine --> MetaKS
Engine --> DataNsKS
Engine --> MetaNsKS
```
### 双轨存储分区与零前缀
- **双轨 4 分区存储结构**：
- **默认库轨道 (`data` / `meta`)**：面向单机单库（`default` / `db 0`）场景，数据键直接使用原始字节存储，物理前缀占用 0 字节；复合结构元数据与子键通过 1 字节键标签分离。
- **租户隔离轨道 (`data_ns` / `meta_ns`)**：面向多租户与多 DB 场景，业务数据落入 `data_ns` 并携带租户编码前缀，租户元数据与 Catalog 目录落入 `meta_ns`，在物理存储层面杜绝键冲突。
- **1 字节紧凑标签 (`KeyTag`)**：复合结构元数据与子键前缀采用 `#[repr(u8)] KeyTag` 编码（如 `\x01[key]`），减少字符串标签开销。
### 保序变长整型编码
- 数据库编号与租户数字 ID 采用 **OPPV 保序变长整型** 编码：
- 数值 $0 \sim 127$ 仅占用 1 字节（相比固定 8 字节大端序降低存储占用）。
- 编码后的字节序与原始数值大小顺序严格一致：$\forall a < b \implies \text{encode}(a) < \text{encode}(b)$，支持直接进行底层范围扫描。
### 租户数字映射与原子重命名
- 租户名称与数字 ID (`ns_id: u64`) 通过全局元数据维护双向映射。
- **租户重命名 (`rename_namespace`)**：仅需原子更新元数据映射关系，无需重写底层业务键。
### 内存友好流式迭代
- **`WeDb::iter(&self) -> Namespaces`**：基于元数据前缀流式扫描租户列表，辅助内存占用为 $O(1)$。
- **`Namespace::iter(&self) -> Dbs`**：直接流式解析 Catalog 目录中编码的激活数据库编号。
---
## 运行时生态与线程模型设计
`wedb_embed` 的底层架构专门针对 `compio` 等基于 Linux `io_uring` 的**一线程一核心**异步运行时进行协同设计，全面发挥单核心独占、无共享状态与物理绑核的硬件局部性优势。
```mermaid
graph LR
subgraph CompioModel["compio 线程模型（一线程一核心）"]
direction TB
C1["CPU 核心 0 (工作线程 0)<br/>绑定物理核心"] --> S1["本地栈缓冲 SmallKey (64B)<br/>L1/L2 缓存热命中 (0 跨核失效)"]
C2["CPU 核心 1 (工作线程 1)<br/>绑定物理核心"] --> S2["本地栈缓冲 SmallKey (64B)<br/>L1/L2 缓存热命中 (0 跨核失效)"]
S1 --> IO1["直接同步调用 / io_uring<br/>无工作窃取 ｜ 无上下文切换"]
S2 --> IO2["直接同步调用 / io_uring<br/>无工作窃取 ｜ 无上下文切换"]
end
subgraph TokioModel["Tokio 传统工作窃取模型"]
direction TB
T1["工作线程 A"] <-->|"跨核任务窃取<br/>L1/L2 缓存颠簸 ｜ NUMA 节点跳转"| T2["工作线程 B"]
T1 --> ST["堆分配状态机 (Send + 'static)<br/>互斥锁竞争 ｜ 破坏栈生命周期"]
T2 --> SB["阻塞线程池切换<br/>线程上下文切换 ｜ 延迟放大 5~10x"]
end
```
### 一线程一核心架构设计
- **栈上生命周期与零堆分配**：<br>
物理键构建广泛采用 `SmallKey` 64 字节栈缓冲与 `SubkeyComposer` 前缀内存复用。在一线程一核心模型下，执行上下文严格局限在单个 CPU 核心的栈帧内，无需向全局堆内存分配器（如 `jemalloc` 或 `glibc malloc`）申请内存，彻底消除了多线程环境下的全局堆分配器互斥锁竞争。
- **CPU 缓存行局部性与零颠簸**：<br>
工作线程与物理 CPU 核心静态绑定，任务执行过程中不发生跨核迁移。热点数据结构（LSM-Tree 内存表索引、布隆过滤器位图、Catalog 元数据缓存）常驻于当前 CPU 的 L1/L2 数据缓存中，避免了 MESI 缓存一致性协议在多核心之间广播无效化（Invalidate）消息引发的缓存行反弹。
- **无锁与轻量并发元数据**：<br>
命名空间与租户目录采用 `papaya::HashMap` 无锁并发哈希表管理，读操作全链路无等待；极低频的元数据写操作采用 `parking_lot` 自适应自旋锁，在单核独占环境下自旋立即完成，不触发内核态 Futex 上下文挂起与跨核线程唤醒。
- **同步内嵌调用与无跨线程调度**：<br>
所有存储引擎 API 均为纯同步内存/磁盘直接调用。在 `compio` 驱动的单线程事件循环中，微秒级与纳秒级的内存查找直接就地完成，与底层 `io_uring` 完成驱动的异步 I/O 驱动无缝协作，避免了将 Future 包装为跨线程状态机的运行时负担。
### 传统多线程工作窃取运行时问题分析
若在 `tokio` 等基于多线程工作窃取与 `epoll` 反应堆的通用异步运行时中使用，会导致以下性能瓶颈与物理损耗：
- **跨核心任务窃取导致缓存失效**：<br>
调度器会在工作线程空闲时跨核窃取任务。同一个请求的 Future 在 `await` 恢复后可能被调度到不同的 CPU 核心或跨 NUMA 节点执行，导致 L1/L2 数据缓存和指令缓存全量失效，引发 CPU 访存延迟抖动与 P99 尾部延迟劣化。
- **破坏零堆分配约束与生命周期提升**：<br>
通用异步任务要求满足 `Send + 'static` 约束，这意味着栈上分配的短期借用结构（如 `&[u8]` 切片、栈上 `SmallKey`）无法跨 `await` 点存活，被迫将键值与临时状态重新包装分配至堆内存（`Box` / `Arc` / `Vec<u8>`），破坏了进程内嵌入式调用的零堆分配优势。
- **事件循环阻塞与阻塞线程池切换开销**：<br>
在基于 `epoll` 的多线程异步运行时中，直接在工作线程执行同步磁盘读取或 CPU 密集型索引查找会阻塞整个事件循环，导致当前线程承载的其他网络连接出现停顿；若通过任务调度将操作转发至阻塞线程池，则会引入线程上下文切换、跨线程通道传递与二次任务调度，导致微秒级进程内操作延迟放大 5~10 倍。
- **多核心内存总线争用与锁冲突**：<br>
在多线程随机争抢共享实例时，跨核心的高频原子操作和互斥锁争用会导致 CPU 内存总线锁定与锁冲突，大幅降低并发吞吐上限。
---
## 多租户与分库隔离
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/multitenant_demo", [])?;
// 获取租户隔离句柄
let ns_apple = db.namespace("tenant_apple");
let ns_google = db.namespace("tenant_google");
ns_apple.set(b"config:theme", b"dark", &[])?;
ns_google.set(b"config:theme", b"light", &[])?;
// 物理隔离读取验证
assert_eq!(ns_apple.get(b"config:theme")?.unwrap(), b"dark");
assert_eq!(ns_google.get(b"config:theme")?.unwrap(), b"light");
// 多分库选择
let db1 = db.select_db(1)?;
db1.set(b"db1_key", b"value1", &[])?;
// 遍历租户下已激活的数据库编号
for db_idx in &ns_apple {
println!("Active DB: {}", db_idx);
}
// 租户重命名
db.rename_namespace("tenant_apple", "tenant_apple_v2")?;
// 键空间管理与清理
let count = ns_apple.key_count()?;
let keys = ns_apple.keys("config:*")?;
let exists = ns_apple.exists(&[b"config:theme"])?;
ns_apple.del(&[b"config:theme"])?;
ns_apple.clear()?; // 级联清空整个租户数据
Ok(())
}
```
---
## 数据结构与接口演示
### 初始化与配置
```rust
use wedb_embed::{Conf, PersistMode, Result, WeDb};
fn main() -> Result<()> {
// 传入配置枚举列表打开数据库（默认开启 LZ4 压缩）
let db = WeDb::open(
"./data/db_demo",
[
Conf::CacheSize(128 * 1024 * 1024), // 128MB 块缓存
Conf::ManualJournalPersist(false), // 启用后台自动刷新预写日志
Conf::WorkerThreads(2), // 后台工作线程数
],
)?;
// 刷盘与周期任务
db.persist(PersistMode::SyncAll)?; // 同步持久化预写日志与脏页
db.active_expire_cycle(100)?; // 触发主动过期采样清理
Ok(())
}
```
### 键值与字符串
```rust
use wedb_embed::{prelude::*, string::Set, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/kv_demo", [])?;
// 基础读写与过期时间
db.set(b"site", b"webc.site", &[])?;
let val = db.get(b"site")?;
db.setex(b"temp_token", b"xyz", 3600_000)?; // 设置毫秒过期时间
// 条件写入
db.set(b"lock", b"1", &[Set::Nx])?;
// 数值自增自减
db.incr(b"counter")?;
db.incrby(b"counter", 10)?;
db.decrby(b"counter", 5)?;
// 批量读写与原子操作
db.mset(&[(b"k1", b"v1"), (b"k2", b"v2")])?;
let values = db.mget(&[b"k1", b"k2"])?;
let len = db.strlen(b"site")?;
let old_val = db.getset(b"site", b"new_site")?;
let del_val = db.getdel(b"site")?;
db.cas(b"site", b"new_site", b"final_site", 0)?; // 比较并交换
Ok(())
}
```
### 哈希与字段级过期
```rust
use wedb_embed::{hash::HExpire, prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/hash_demo", [])?;
// 字段写入与读取
db.hset(b"user:100", &[(b"name", b"Alice"), (b"age", b"20")])?;
let name = db.hget(b"user:100", b"name")?;
// 批量写入与获取
db.hmset(b"user:100", &[(b"city", b"Beijing"), (b"role", b"Admin")])?;
let fields = db.hmget(b"user:100", &[b"name", b"city"])?;
// 数值自增与元数据
db.hincrby(b"user:100", b"age", 1)?;
let exists = db.hexists(b"user:100", b"name")?;
let len = db.hlen(b"user:100")?;
let all = db.hgetall(b"user:100")?;
let keys = db.hkeys(b"user:100")?;
let vals = db.hvals(b"user:100")?;
db.hdel(b"user:100", &[b"role"])?;
// 字段级独立过期时间
db.hexpire(b"user:100", &[b"name"], 3600, HExpire::None)?;
let ttls = db.httl(b"user:100", &[b"name"])?;
Ok(())
}
```
### 列表
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/list_demo", [])?;
// 双端推入与弹出
db.lpush(b"tasks", &[b"task1", b"task2"])?;
db.rpush(b"tasks", &[b"task3"])?;
let first = db.lpop_one(b"tasks")?;
let last = db.rpop_one(b"tasks")?;
// 范围查询、修剪与修改
let len = db.llen(b"tasks")?;
let items = db.lrange(b"tasks", 0, -1)?;
db.ltrim(b"tasks", 0, 10)?;
db.lset(b"tasks", 0, b"updated_task")?;
Ok(())
}
```
### 集合
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/set_demo", [])?;
db.sadd(b"tags", &[b"rust", b"database", b"lsm"])?;
let is_member = db.sismember(b"tags", b"rust")?;
let count = db.scard(b"tags")?;
let members = db.smembers(b"tags")?;
db.srem(b"tags", &[b"lsm"])?;
let popped = db.spop(b"tags", 1)?;
// 集合代数运算
db.sadd(b"tags_other", &[b"rust", b"storage"])?;
let inter = db.sinter(&[b"tags", b"tags_other"])?;
let union = db.sunion(&[b"tags", b"tags_other"])?;
let diff = db.sdiff(&[b"tags", b"tags_other"])?;
Ok(())
}
```
### 有序集合
```rust
use wedb_embed::{prelude::*, zset::RangeScoreSpec, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/zset_demo", [])?;
// 添加成员与分数
db.zadd(b"leaderboard", &[(100.0, b"player1"), (200.0, b"player2")], &[])?;
db.zincrby(b"leaderboard", 50.0, b"player1")?;
// 排名与分数查询
let score = db.zscore(b"leaderboard", b"player1")?;
let rank = db.zrank(b"leaderboard", b"player1")?;
let rev_rank = db.zrevrank(b"leaderboard", b"player1")?;
// 范围检索与计数
let top = db.zrange(b"leaderboard", 0, 10)?;
let count = db.zcount(b"leaderboard", &RangeScoreSpec::new(100.0, 300.0))?;
let card = db.zcard(b"leaderboard")?;
db.zrem(b"leaderboard", &[b"player1"])?;
Ok(())
}
```
### 位图与位域
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/bitmap_demo", [])?;
// 位设置与统计
db.setbit(b"online_users", 1001, 1)?;
let is_online = db.getbit(b"online_users", 1001)?;
let online_count = db.bitcount(b"online_users", None, None)?;
let first_online = db.bitpos(b"online_users", 1, None, None)?;
Ok(())
}
```
### JSON 与路径查询
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/json_demo", [])?;
// JSON 写入与 JSONPath 表达式查询
db.json_set(b"doc:1", "$", r#"{"user":{"name":"Alice","age":25,"roles":["admin"]}}"#)?;
let name = db.json_get(b"doc:1", Some("$.user.name"))?;
db.json_numincrby(b"doc:1", "$.user.age", "1.0")?;
db.json_arrappend(b"doc:1", "$.user.roles", &[r#""editor""#])?;
Ok(())
}
```
### 布隆与布谷鸟过滤器
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/filter_demo", [])?;
// 布隆过滤器
db.bf_reserve(b"bf_filter", 0.01, 10000, 2)?;
db.bf_add(b"bf_filter", b"item_1")?;
let exists = db.bf_exists(b"bf_filter", b"item_1")?;
// 布谷鸟过滤器
db.cf_reserve(b"cf_filter", 10000, 2, 500, 1)?;
db.cf_add(b"cf_filter", b"item_2")?;
let cf_exists = db.cf_exists(b"cf_filter", b"item_2")?;
db.cf_del(b"cf_filter", b"item_2")?;
Ok(())
}
```
### 时序数据与聚合
```rust
use wedb_embed::{
prelude::*,
timeseries::{AggregationType, DuplicatePolicy, TSRangeOption},
Result, WeDb,
};
fn main() -> Result<()> {
let db = WeDb::open("./data/timeseries_demo", [])?;
// 创建时序序列
db.ts_create(b"cpu:usage", 0, DuplicatePolicy::Last, &[("host", "server-1")])?;
db.ts_add(b"cpu:usage", 1700000000000, 42.5)?;
let latest = db.ts_get(b"cpu:usage")?;
// 窗口降采样范围查询
let samples = db.ts_range_opt(
b"cpu:usage",
&TSRangeOption {
from_timestamp: 0,
to_timestamp: 1800000000000,
aggregation: Some((AggregationType::Avg, 60000)),
..Default::default()
},
)?;
Ok(())
}
```
### 地理空间位置
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/geo_demo", [])?;
// 坐标添加与距离计算
db.geoadd(
b"cities",
&[
(116.4074, 39.9042, b"Beijing"),
(121.4737, 31.2304, b"Shanghai"),
],
)?;
let dist_km = db.geodist(b"cities", b"Beijing", b"Shanghai", Some("km"))?;
let hash = db.geohash(b"cities", &[b"Beijing"])?;
Ok(())
}
```
### 基数统计
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/hll_demo", [])?;
db.pfadd(b"hll_uv", &[b"user_1", b"user_2", b"user_3"])?;
let uv_count = db.pfcount(&[b"hll_uv"])?;
db.pfmerge(b"hll_merged", &[b"hll_uv"])?;
Ok(())
}
```
### 分位数统计
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/tdigest_demo", [])?;
db.tdigest_create(b"latencies", 100.0)?;
db.tdigest_add(b"latencies", &[12.5, 15.0, 18.2, 45.0, 99.9])?;
let p95 = db.tdigest_quantile(b"latencies", &[0.95])?;
let cdf = db.tdigest_cdf(b"latencies", &[20.0])?;
Ok(())
}
```
### 紧凑整型集合
```rust
use wedb_embed::{prelude::*, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/sortedint_demo", [])?;
// 64 位紧凑整型集合
db.si_add(b"post:100:likes", &[1001, 1002, 1003, 1004])?;
let has_liked = db.si_exists(b"post:100:likes", 1001)?;
let like_count = db.si_card(b"post:100:likes")?;
let top_likers = db.si_range(b"post:100:likes", 0, 0, 10, false)?;
Ok(())
}
```
### 消息流与消费组
```rust
use wedb_embed::{prelude::*, stream::StreamId, Result, WeDb};
fn main() -> Result<()> {
let db = WeDb::open("./data/stream_demo", [])?;
// 追加消息
let entry_id = db.xadd_simple(
b"event_stream",
None,
&[(b"action", b"login"), (b"uid", b"1001")],
)?;
let len = db.xlen(b"event_stream")?;
let entries = db.xrange(b"event_stream", StreamId::MIN, StreamId::MAX, Some(10))?;
// 消费组与待处理条目统计
db.xgroup_create(b"event_stream", "group_workers", "0-0", false, None)?;
let pending = db.xpending_summary(b"event_stream", "group_workers")?;
Ok(())
}
```
### 全文检索与向量检索
```rust
use wedb_embed::{
prelude::*,
search::{
DistanceMetric, FtCreate, FtSearch, IndexField, IndexFieldType, IndexOnDataType,
SearchIndexManager, VectorAlgorithm, VectorFieldMetadata, VectorType,
},
Result, WeDb,
};
fn main() -> Result<()> {
let db = WeDb::open("./data/search_demo", [])?;
let mut search_mgr = SearchIndexManager::new();
// 创建多字段模式
let mut schema = FtCreate::new("idx_articles", IndexOnDataType::Json);
schema.prefixes.push("article:".to_string());
schema.fields.push(IndexField {
name: "title".to_string(),
alias: None,
field_type: IndexFieldType::Text,
weight: 1.0,
sortable: false,
noindex: false,
separator: None,
casesensitive: false,
withsuffixtrie: false,
unf: false,
vector_meta: None,
});
schema.fields.push(IndexField {
name: "vec".to_string(),
alias: None,
field_type: IndexFieldType::Vector,
weight: 1.0,
sortable: false,
noindex: false,
separator: None,
casesensitive: false,
withsuffixtrie: false,
unf: false,
vector_meta: Some(VectorFieldMetadata {
algorithm: VectorAlgorithm::Hnsw,
vector_type: VectorType::Float32,
dim: 4,
distance_metric: DistanceMetric::Cosine,
m: 16,
ef_construction: 200,
ef_runtime: 10,
block_size: 1024,
initial_cap: 1000,
}),
});
search_mgr.create_index_from_opts(schema)?;
// 文本搜索与向量近邻检索
let res = search_mgr.search("idx_articles", "@title:database", &FtSearch::default())?;
println!("Found matching docs: {}", res.total);
Ok(())
}
```
---
## 技术堆栈
- **开发语言**：Rust Edition 2024
- **存储引擎**：`fjall` 分层存储持久化引擎
- **JSON 引擎**：`sonic-rs` SIMD 指令集解析
- **非加密哈希**：`rapidhash` 高效哈希算法
- **序列化编解码**：`bitcode` 紧凑二进制序列化
- **字符串与内存**：`hipstr` 紧凑字符串存储与零拷贝借用
- **并发哈希表**：`papaya` 无锁并发哈希表
- **集合与位运算**：`roaring`、`memchr`、`crc32fast`、`fastrand`
- **时间戳处理**：`coarsetime`、`ts_`
- **数值与浮点序列化**：`zmij`、`itoa`
- **枚举派生**：`strum`
- **错误管理**：`thiserror`