jdb_proto 0.1.2

[English](#en) | [中文](#zh)

---

<a id="en"></a>

# jdb_proto : System Core Protocol Layer

High-performance protocol definitions for distributed database communication with zero-copy memory optimization.

## Table of Contents

- [Project Overview](#project-overview)
- [Features](#features)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
- [Design Architecture](#design-architecture)
- [Technical Stack](#technical-stack)
- [Project Structure](#project-structure)
- [Historical Context](#historical-context)

## Project Overview

jdb_proto provides the core protocol layer for the jdb distributed database system. It defines essential data structures, communication protocols, and unified error handling mechanisms for high-performance database operations.

The library implements a WiscKey-style storage architecture with zero-copy memory optimization, enabling efficient data access patterns in distributed environments.

## Features

- **Zero-copy Memory Optimization**: Leverages `Bytes` for efficient memory management
- **WiscKey Storage Pattern**: Separates keys from values for optimal disk usage
- **Async Communication**: Built on tokio with high-performance async channels
- **Type Safety**: Compile-time guarantees for memory layout and data integrity
- **Unified Error Handling**: Comprehensive error propagation across system layers

## Quick Start

Add to your `Cargo.toml`:

```toml
[dependencies]
jdb_proto = "0.1.1"
tokio = { version = "1.48", features = ["full"] }
```

Basic usage example:

```rust
use jdb_proto::*;
use bytes::Bytes;

#[tokio::main]
async fn main() -> Result<()> {
  // Create communication channels
  let (tx, rx) = new_shard_channel(1000);
  
  // Create a request
  let (resp_tx, resp_rx) = new_resp_channel();
  let req = Req::Set {
    key: Bytes::from("hello"),
    val: Bytes::from("world"),
    resp: resp_tx,
  };
  
  // Send request (in real implementation)
  tx.send(req).await?;
  
  Ok(())
}
```

## API Reference

### Core Data Types

#### `Key` and `Val`
```rust
pub type Key = Bytes;
pub type Val = Bytes;
```
Immutable byte sequences with zero-copy reference counting.

#### `ValPtr`
```rust
pub struct ValPtr {
  pub offset: u64,    // File offset (8 bytes)
  pub file_id: u32,   // Log file ID (4 bytes)
  pub len: u32,       // Data length (4 bytes)
}
```
Logical pointer to disk data with 16-byte fixed memory layout.

### Communication Protocol

#### Request Types
```rust
pub enum Req {
  Set { key: Key, val: Val, resp: RespTx<()> },
  Get { key: Key, resp: RespTx<Option<Val>> },
  Del { key: Key, resp: RespTx<bool> },
}
```

#### Channel Types
- `ShardTx/ShardRx`: Multi-producer, single-consumer channels for shard communication
- `RespTx/RespRx`: One-shot channels for request-response patterns

### Error Handling

```rust
pub enum Error {
  IO(std::io::Error),
  Internal(String),
  Busy,
  Shutdown,
  NotFound,
}
```

## Design Architecture

```mermaid
graph TD
    Client[Client] -->|MPSC| Shard[Shard]
    Shard -->|Oneshot| Client
    
    Shard -->|File I/O| Storage[Storage Layer]
    
    subgraph "Protocol Layer"
        Req[Request Enum]
        ValPtr[Value Pointer]
        Channels[Async Channels]
    end
    
    Client --> Req
    Shard --> Req
    Req --> ValPtr
    Req --> Channels
```

The architecture follows a layered approach:

1. **Protocol Layer**: Defines communication primitives and data structures
2. **Channel Layer**: Handles async message passing between components
3. **Storage Interface**: Provides abstractions for disk operations

### Module Interaction Flow

1. Client creates request using `Req` enum
2. Request sent through `ShardTx` to appropriate shard
3. Shard processes request and sends response via `RespTx`
4. Client receives response through `RespRx`

## Technical Stack

- **Core Language**: Rust (Edition 2024)
- **Async Runtime**: Tokio
- **Memory Management**: `bytes` crate for zero-copy operations
- **Channel Communication**: `crossfire` for high-performance MPSC
- **One-shot Communication**: `async-oneshot` for request-response

## Project Structure

```
src/
├── lib.rs      # Main protocol definitions
└── error.rs    # Unified error handling

tests/
└── main.rs     # Comprehensive test suite
```

### Key Components

- **Data Types**: Zero-copy byte sequences and value pointers
- **Communication**: Async channels with configurable backpressure
- **Error Handling**: Unified error type with automatic conversion

## Historical Context

The WiscKey architecture, which inspired `ValPtr` design, emerged from the University of Wisconsin-Madison in 2016. It challenged traditional LSM-tree designs by separating keys from values, reducing write amplification and improving SSD performance.

This protocol layer builds upon those principles while adding modern async communication patterns inspired by systems like TiKV and CockroachDB. The zero-copy design philosophy traces back to Linux's `sendfile()` system call (2003) and has evolved through modern frameworks like Cap'n Proto and FlatBuffers.

The MPSC channel design draws from Erlang's actor model (1986) and Go's channels (2009), adapted for Rust's ownership system while maintaining the safety guarantees that make Rust unique in systems programming.

---

## About

This project is an open-source component of [js0.site ⋅ Refactoring the Internet Plan](https://js0.site).

We are redefining the development paradigm of the Internet in a componentized way. Welcome to follow us:

* [Google Group](https://groups.google.com/g/js0-site)
* [js0site.bsky.social](https://bsky.app/profile/js0site.bsky.social)

---

<a id="zh"></a>

# jdb_proto : 系统核心协议层

面向分布式数据库通信的高性能协议定义，具备零拷贝内存优化特性。

## 目录

- [项目概述](#项目概述)
- [核心特性](#核心特性)
- [快速开始](#快速开始)
- [API 参考](#api-参考)
- [设计架构](#设计架构)
- [技术堆栈](#技术堆栈)
- [目录结构](#目录结构)
- [历史背景](#历史背景)

## 项目概述

jdb_proto 为 jdb 分布式数据库系统提供核心协议层。它定义了基本数据结构、通信协议和统一错误处理机制，支持高性能数据库操作。

该库采用 WiscKey 风格的存储架构，具备零拷贝内存优化，在分布式环境中实现高效的数据访问模式。

## 核心特性

- **零拷贝内存优化**：利用 `Bytes` 实现高效内存管理
- **WiscKey 存储模式**：分离键值以优化磁盘使用
- **异步通信**：基于 tokio 的高性能异步通道
- **类型安全**：编译时保证内存布局和数据完整性
- **统一错误处理**：跨系统层的综合错误传播

## 快速开始

在 `Cargo.toml` 中添加：

```toml
[dependencies]
jdb_proto = "0.1.1"
tokio = { version = "1.48", features = ["full"] }
```

基础使用示例：

```rust
use jdb_proto::*;
use bytes::Bytes;

#[tokio::main]
async fn main() -> Result<()> {
  // 创建通信通道
  let (tx, rx) = new_shard_channel(1000);
  
  // 创建请求
  let (resp_tx, resp_rx) = new_resp_channel();
  let req = Req::Set {
    key: Bytes::from("hello"),
    val: Bytes::from("world"),
    resp: resp_tx,
  };
  
  // 发送请求（实际实现中）
  tx.send(req).await?;
  
  Ok(())
}
```

## API 参考

### 核心数据类型

#### `Key` 和 `Val`
```rust
pub type Key = Bytes;
pub type Val = Bytes;
```
具备零拷贝引用计数的不可变字节序列。

#### `ValPtr`
```rust
pub struct ValPtr {
  pub offset: u64,    // 文件偏移量 (8 字节)
  pub file_id: u32,   // 日志文件 ID (4 字节)
  pub len: u32,       // 数据长度 (4 字节)
}
```
指向磁盘数据的逻辑指针，固定 16 字节内存布局。

### 通信协议

#### 请求类型
```rust
pub enum Req {
  Set { key: Key, val: Val, resp: RespTx<()> },
  Get { key: Key, resp: RespTx<Option<Val>> },
  Del { key: Key, resp: RespTx<bool> },
}
```

#### 通道类型
- `ShardTx/ShardRx`：用于分片通信的多生产者单消费者通道
- `RespTx/RespRx`：用于请求-响应模式的单次通道

### 错误处理

```rust
pub enum Error {
  IO(std::io::Error),
  Internal(String),
  Busy,
  Shutdown,
  NotFound,
}
```

## 设计架构

```mermaid
graph TD
    Client[客户端] -->|MPSC| Shard[分片]
    Shard -->|Oneshot| Client
    
    Shard -->|文件 I/O| Storage[存储层]
    
    subgraph "协议层"
        Req[请求枚举]
        ValPtr[值指针]
        Channels[异步通道]
    end
    
    Client --> Req
    Shard --> Req
    Req --> ValPtr
    Req --> Channels
```

架构采用分层设计：

1. **协议层**：定义通信原语和数据结构
2. **通道层**：处理组件间异步消息传递
3. **存储接口**：提供磁盘操作抽象

### 模块交互流程

1. 客户端使用 `Req` 枚举创建请求
2. 请求通过 `ShardTx` 发送到相应分片
3. 分片处理请求并通过 `RespTx` 发送响应
4. 客户端通过 `RespRx` 接收响应

## 技术堆栈

- **核心语言**：Rust (Edition 2024)
- **异步运行时**：Tokio
- **内存管理**：`bytes` crate 实现零拷贝操作
- **通道通信**：`crossfire` 实现高性能 MPSC
- **单次通信**：`async-oneshot` 实现请求-响应

## 目录结构

```
src/
├── lib.rs      # 主要协议定义
└── error.rs    # 统一错误处理

tests/
└── main.rs     # 综合测试套件
```

### 关键组件

- **数据类型**：零拷贝字节序列和值指针
- **通信**：可配置背压的异步通道
- **错误处理**：具备自动转换的统一错误类型

## 历史背景

启发 `ValPtr` 设计的 WiscKey 架构诞生于 2016 年的威斯康星大学麦迪逊分校。它挑战了传统的 LSM-tree 设计，通过分离键值来减少写放大并提升 SSD 性能。

此协议层基于这些原则构建，同时融入了受 TiKV 和 CockroachDB 等系统启发的现代异步通信模式。零拷贝设计理念可追溯至 Linux 的 `sendfile()` 系统调用（2003 年），并通过 Cap'n Proto 和 FlatBuffers 等现代框架不断发展。

MPSC 通道设计借鉴自 Erlang 的 Actor 模型（1986 年）和 Go 的通道（2009 年），适配 Rust 的所有权系统，同时保持 Rust 在系统编程中独特的安全性保证。

---

## 关于

本项目为 [js0.site ⋅ 重构互联网计划](https://js0.site) 的开源组件。

我们正在以组件化的方式重新定义互联网的开发范式，欢迎关注：

* [谷歌邮件列表](https://groups.google.com/g/js0-site)
* [js0site.bsky.social](https://bsky.app/profile/js0site.bsky.social)