v_queue 0.3.0

simple file based queue
Documentation
# Quick Reference

Quick reference guide for common V-Queue operations.

## Server Commands

### Start Server

```bash
# Default
v-queue-server

# Custom config
v-queue-server --config /etc/vqueue/server.toml

# Override settings
v-queue-server --bind 0.0.0.0:9093 --data-dir /var/lib/vqueue

# Disable auth
v-queue-server --no-auth

# Debug mode
v-queue-server --log-level debug
```

### Server Management

```bash
# Start
sudo systemctl start v-queue-server

# Stop
sudo systemctl stop v-queue-server

# Restart
sudo systemctl restart v-queue-server

# Status
sudo systemctl status v-queue-server

# Logs
journalctl -u v-queue-server -f
```

## API Endpoints

### Health Check

```bash
curl http://localhost:9093/health
```

### List Queues

```bash
curl -u admin:password http://localhost:9093/api/v1/queues
```

### Queue Info

```bash
curl -u admin:password http://localhost:9093/api/v1/queues/{queue}
```

### List Consumers

```bash
curl -u admin:password http://localhost:9093/api/v1/queues/{queue}/consumers
```

### Consume Messages

```bash
# Basic
curl -u admin:password \
  "http://localhost:9093/api/v1/queues/{queue}/consumers/{consumer}/messages"

# With timeout and limit
curl -u admin:password \
  "http://localhost:9093/api/v1/queues/{queue}/consumers/{consumer}/messages?timeout_ms=30000&max_messages=100"
```

### Commit Consumer Position

```bash
curl -X POST -u admin:password \
  http://localhost:9093/api/v1/queues/{queue}/consumers/{consumer}/commit
```

## Python Client

### Basic Consumer

```python
import requests
from requests.auth import HTTPBasicAuth

auth = HTTPBasicAuth("admin", "password")

# Consume
response = requests.get(
    "http://localhost:9093/api/v1/queues/events/consumers/my-app/messages",
    params={"timeout_ms": 30000, "max_messages": 100},
    auth=auth
)
messages = response.json()["messages"]

# Process
for msg in messages:
    print(f"Offset: {msg['offset']}, Value: {msg['value']}")

# Commit
requests.post(
    "http://localhost:9093/api/v1/queues/events/consumers/my-app/commit",
    auth=auth
)
```

## Node.js Client

### Basic Consumer

```javascript
const axios = require('axios');

const client = axios.create({
  baseURL: 'http://localhost:9093/api/v1',
  auth: { username: 'admin', password: 'password' }
});

// Consume
const response = await client.get(
  '/queues/events/consumers/my-app/messages',
  { params: { timeout_ms: 30000, max_messages: 100 } }
);
const messages = response.data.messages;

// Process
for (const msg of messages) {
  console.log(`Offset: ${msg.offset}, Value: ${msg.value}`);
}

// Commit
await client.post('/queues/events/consumers/my-app/commit');
```

## Configuration

### Basic Config

```toml
# v-queue-server.toml
bind_address = "127.0.0.1:9093"
data_directory = "./queues"
log_level = "info"
max_message_size = 1048576
default_batch_size = 100
auth_enabled = false
```

### With Authentication

```toml
bind_address = "0.0.0.0:9093"
data_directory = "/var/lib/vqueue/queues"
log_level = "warn"
auth_enabled = true

[users]
admin = "secure_password"
app1 = "app1_password"
```

## File Structure

```
/var/lib/vqueue/queues/
├── events_info_queue              # Queue metadata
├── events_queue.lock              # Write lock
├── events-0/                      # Partition 0
│   ├── events_queue               # Message data
│   └── events_info_push           # Write position
└── events_info_pop_consumer1      # Consumer offset
```

## Common Tasks

### Reset Consumer Position

```bash
# Delete consumer offset file
rm /var/lib/vqueue/queues/{queue}_info_pop_{consumer}

# Next consume starts from beginning
```

### Check Queue Size

```bash
du -sh /var/lib/vqueue/queues/{queue}-*
```

### Monitor Queue Growth

```bash
watch -n 5 'du -sh /var/lib/vqueue/queues/*'
```

### Backup Queue Data

```bash
cp -r /var/lib/vqueue/queues /backup/vqueue-$(date +%Y%m%d)
```

### Clean Old Partitions

```bash
# Manually remove old partition (ensure consumers don't need it)
rm -rf /var/lib/vqueue/queues/{queue}-0/
```

## Performance Tuning

### Mount Options

```bash
# /etc/fstab
/dev/sda1 /var/lib/vqueue ext4 noatime,nodiratime 0 2
```

### File Limits

```bash
# /etc/security/limits.conf
vqueue soft nofile 65536
vqueue hard nofile 65536
```

### System Tuning

```bash
# /etc/sysctl.conf
fs.file-max = 2097152
net.core.somaxconn = 1024
```

## Troubleshooting

### Check Server Status

```bash
# Process running?
ps aux | grep v-queue-server

# Port listening?
netstat -tlnp | grep 9093

# Disk space?
df -h /var/lib/vqueue

# Permissions?
ls -la /var/lib/vqueue/queues
```

### Common Fixes

```bash
# Fix permissions
sudo chown -R vqueue:vqueue /var/lib/vqueue
sudo chmod -R 755 /var/lib/vqueue

# Remove stale lock
rm /var/lib/vqueue/queues/{queue}_queue.lock

# Increase file descriptors
ulimit -n 65536
```

### Debug Mode

```bash
# Enable debug logging
v-queue-server --log-level debug

# Watch logs
journalctl -u v-queue-server -f
```

## Useful Commands

### List All Consumers

```bash
ls -1 /var/lib/vqueue/queues/*_*_info | \
  sed 's/.*\///' | \
  sed 's/_info$//' | \
  sed 's/.*_//'
```

### Find Queue with Most Data

```bash
du -sh /var/lib/vqueue/queues/*-* | sort -h | tail -n 5
```

### Count Messages in Partition

```bash
# Approximate (file size / average message size)
stat -c%s /var/lib/vqueue/queues/{queue}-0/{queue}_queue
```

### Monitor Disk I/O

```bash
iostat -x 1
```

### Monitor Network

```bash
netstat -anp | grep 9093 | wc -l  # Active connections
```

## Environment Variables (for clients)

```bash
# Bash
export VQUEUE_URL="http://localhost:9093"
export VQUEUE_USER="admin"
export VQUEUE_PASSWORD="password"

# Python
import os
url = os.getenv("VQUEUE_URL", "http://localhost:9093")
user = os.getenv("VQUEUE_USER")
password = os.getenv("VQUEUE_PASSWORD")
```

## Docker

### Build Image

```bash
docker build -t v-queue-server .
```

### Run Container

```bash
docker run -d \
  -p 9093:9093 \
  -v /path/to/data:/var/lib/vqueue/queues \
  v-queue-server
```

### Docker Compose

```yaml
version: '3.8'
services:
  v-queue:
    build: .
    ports:
      - "9093:9093"
    volumes:
      - ./queues:/var/lib/vqueue/queues
    restart: unless-stopped
```

## Quick Links

- [Full Documentation]README.md
- [API Reference]05-api-reference.md
- [Client Examples]07-client-examples.md
- [Troubleshooting]09-troubleshooting.md