v_queue 0.3.0

simple file based queue
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

# V-Queue Overview

V-Queue is a simple, high-performance file-based message queue system implemented in Rust.

## What is V-Queue?

V-Queue is a message queue system designed for reliability and simplicity. It stores messages directly on disk using sequential file I/O, providing high throughput while ensuring data durability.

## Key Features

### Core Library (v-queue)

- **High-Performance Sequential I/O**: Optimized for fast disk operations
- **Data Integrity**: CRC32 checksums on all messages
- **Automatic Partitioning**: Manages multiple file partitions automatically
- **Consumer Offset Management**: Tracks consumer positions in queues
- **Thread-Safe**: Safe concurrent access from multiple threads
- **Java Bindings**: JNI support for Java applications
- **Zero External Dependencies**: Minimal dependency footprint

### Network Server (v-queue-server)

- **RESTful HTTP API**: Consumer API for reading messages
- **Consumer Model**: HTTP endpoints for message consumption
- **Multi-Consumer Support**: Multiple consumers per queue
- **Basic Authentication**: Optional security layer
- **Long Polling**: Efficient message consumption with timeouts
- **JSON and Binary Messages**: Flexible message formats
- **Health Monitoring**: Built-in health check endpoint
- **Cross-Platform**: Runs on Linux, macOS, Windows
- **Note**: Message production requires direct library access (no HTTP producer API)

## Architecture Components

```
┌─────────────────────────────────────────────────────┐
│                   HTTP Clients                       │
│          (Python, Node.js, Rust, curl)              │
└─────────────────────┬───────────────────────────────┘
                      │ REST API
┌─────────────────────▼───────────────────────────────┐
│              V-Queue Server (Axum)                   │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────┐│
│  │     API      │  │     Auth     │  │   Queue    ││
│  │   Handlers   │  │  Middleware  │  │  Manager   ││
│  └──────────────┘  └──────────────┘  └────────────┘│
└─────────────────────┬───────────────────────────────┘
                      │ Direct Access
┌─────────────────────▼───────────────────────────────┐
│              Core V-Queue Library                    │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────┐│
│  │    Queue     │  │   Consumer   │  │   Record   ││
│  │   Manager    │  │   Tracking   │  │   Format   ││
│  └──────────────┘  └──────────────┘  └────────────┘│
└─────────────────────┬───────────────────────────────┘
                      │ File I/O
┌─────────────────────▼───────────────────────────────┐
│                File System Storage                   │
│              (Partitioned Queue Files)               │
└─────────────────────────────────────────────────────┘
```

## Use Cases

### Log Aggregation
Collect logs from multiple services and process them asynchronously.

### Event Processing
Distribute events to multiple consumers for parallel processing.

### Task Queues
Implement background job processing with reliable delivery.

### Data Pipelines
Build data processing pipelines with guaranteed message ordering.

### Message Buffering
Buffer messages between services with different processing speeds.

## Comparison with Other Solutions

| Feature | V-Queue | Kafka | RabbitMQ | Redis Streams |
|---------|---------|-------|----------|---------------|
| Setup Complexity | Low | High | Medium | Low |
| Dependencies | None | JVM, ZooKeeper | Erlang | Redis |
| Persistence | Always | Configurable | Configurable | Configurable |
| Performance | High | Very High | Medium | High |
| Ordering | Guaranteed | Guaranteed | Optional | Guaranteed |
| Multi-consumer | Yes | Yes | Yes | Yes |
| HTTP API | Yes | No (needs proxy) | Optional | No |
| Resource Usage | Low | High | Medium | Medium |

## Design Principles

1. **Simplicity**: Easy to deploy, configure, and operate
2. **Reliability**: All data persisted to disk with integrity checks
3. **Performance**: Optimized for sequential I/O patterns
4. **No External Dependencies**: Standalone binary
5. **Standard Protocols**: HTTP/REST for maximum compatibility
6. **File-Based Storage**: Simple to backup, replicate, and debug

## Limitations

- **Single-node**: No built-in clustering or replication
- **Local Disk Only**: Requires local file system
- **Sequential Processing**: Messages processed in order per consumer
- **No Message Filtering**: Consumers receive all messages
- **Basic Security**: Only HTTP Basic Authentication

## Next Steps

- [Architecture Details](02-architecture.md)
- [Installation Guide](03-installation.md)
- [API Reference](05-api-reference.md)