v_queue 0.3.0

# Troubleshooting Guide

Common issues and solutions for V-Queue.

## Connection Issues

### Issue: Cannot connect to server

**Symptoms**:
```bash
curl: (7) Failed to connect to localhost port 9093: Connection refused
```

**Causes**:
- Server not running
- Wrong port number
- Firewall blocking connection
- Server bound to different address

**Solutions**:

1. **Check if server is running**:
```bash
ps aux | grep v-queue-server
systemctl status v-queue-server
```

2. **Check server logs**:
```bash
journalctl -u v-queue-server -f
```

3. **Verify bind address**:
```bash
netstat -tlnp | grep 9093
# or
ss -tlnp | grep 9093
```

4. **Check firewall**:
```bash
sudo ufw status
sudo iptables -L -n
```

5. **Try different bind address**:
```bash
v-queue-server --bind 0.0.0.0:9093
```

### Issue: Connection timeout

**Symptoms**:
```python
requests.exceptions.Timeout: Request timed out
```

**Causes**:
- Network issues
- Server overloaded
- Client timeout too short

**Solutions**:

1. **Increase client timeout**:
```python
response = requests.get(url, timeout=60)  # 60 seconds
```

2. **Check server load**:
```bash
top
htop
```

3. **Check network latency**:
```bash
ping your-server
traceroute your-server
```

## Authentication Issues

### Issue: 401 Unauthorized

**Symptoms**:
```json
{"error": "Unauthorized"}
```

**Causes**:
- Wrong username or password
- Authentication not configured
- Credentials not sent

**Solutions**:

1. **Verify credentials**:
```bash
# Check config file
cat v-queue-server.toml | grep -A5 "\[users\]"
```

2. **Test with correct credentials**:
```bash
curl -u admin:password http://localhost:9093/api/v1/queues
```

3. **Check if auth is enabled**:
```toml
# In config file
auth_enabled = true  # Should be true
```

4. **Disable auth for testing**:
```bash
v-queue-server --no-auth
```

### Issue: Authentication configured but not working

**Symptoms**:
- 401 errors even with correct password
- Changes to config not taking effect

**Causes**:
- Server not restarted after config change
- Wrong config file being used

**Solutions**:

1. **Restart server**:
```bash
sudo systemctl restart v-queue-server
```

2. **Verify config file location**:
```bash
v-queue-server --config /path/to/config.toml
```

3. **Check logs for config load errors**:
```bash
journalctl -u v-queue-server | grep -i config
```

## Queue Issues

### Issue: Queue not found (404)

**Symptoms**:
```json
{"error": "Queue not found: my-queue"}
```

**Causes**:
- Queue doesn't exist yet
- Wrong queue name
- Permissions issue

**Solutions**:

1. **List available queues**:
```bash
curl -u admin:password http://localhost:9093/api/v1/queues
```

2. **Check queue directory**:
```bash
ls -la /var/lib/vqueue/queues/
```

3. **Queues must be created by writing messages** - The HTTP server provides consumer-only API. Queues are created when first message is written using the v_queue library directly (via Rust code or JNI bindings). They cannot be created via HTTP API.

4. **Check permissions**:
```bash
ls -la /var/lib/vqueue/queues/
sudo chown -R vqueue:vqueue /var/lib/vqueue/queues/
```

### Issue: Queue lock error

**Symptoms**:
```
Error: AlreadyOpen
```

**Causes**:
- Another process has queue open for writing
- Stale lock file from crash
- Multiple writers to same queue

**Solutions**:

1. **Check for running processes**:
```bash
lsof /var/lib/vqueue/queues/my-queue_queue.lock
```

2. **Remove stale lock** (if no process using it):
```bash
rm /var/lib/vqueue/queues/my-queue_queue.lock
```

3. **Only one writer allowed per queue** - this is by design

### Issue: Cannot read messages

**Symptoms**:
- Empty messages array returned
- Consumer stuck at same position

**Causes**:
- Queue is empty
- Consumer already at end of queue
- Corrupted consumer offset file

**Solutions**:

1. **Check queue info**:
```bash
curl -u admin:password http://localhost:9093/api/v1/queues/my-queue
```

2. **Check if messages exist**:
```bash
ls -lh /var/lib/vqueue/queues/my-queue-0/my-queue_queue
# Non-zero size means messages exist
```

3. **Reset consumer position** (delete consumer offset):
```bash
rm /var/lib/vqueue/queues/my-queue_info_pop_my-consumer
```

Next consume will start from beginning.

## Performance Issues

### Issue: Slow message consumption

**Symptoms**:
- High latency on consume requests
- Low throughput

**Causes**:
- Small batch size
- Slow disk
- Network latency

**Solutions**:

1. **Increase batch size**:
```bash
curl "http://localhost:9093/api/v1/queues/events/consumers/my-consumer/messages?max_messages=1000"
```

2. **Check disk performance**:
```bash
iostat -x 1
```

3. **Use SSD instead of HDD**

4. **Reduce network overhead** - run consumer closer to server

### Issue: High CPU usage

**Symptoms**:
- v-queue-server using high CPU
- System slow

**Causes**:
- Debug logging enabled
- Many concurrent requests
- Inefficient polling (timeout=0)

**Solutions**:

1. **Reduce log level**:
```toml
log_level = "warn"
```

2. **Use longer timeouts**:
```python
messages = consumer.consume("events", "consumer", timeout=30)
```

3. **Reduce concurrent connections**

4. **Check for loops without delays**:
```python
# Bad
while True:
    messages = consume(timeout=0)

# Good  
while True:
    messages = consume(timeout=30)
    time.sleep(1)
```

### Issue: Disk space filling up

**Symptoms**:
- Queue directory growing large
- Disk space warnings

**Causes**:
- No retention policy (v-queue keeps all messages)
- Consumers not keeping up
- Large message sizes

**Solutions**:

1. **Check queue sizes**:
```bash
du -sh /var/lib/vqueue/queues/*
```

2. **Manually clean old partitions** (if consumers caught up):
```bash
# Find old partitions
ls -la /var/lib/vqueue/queues/events-0/
ls -la /var/lib/vqueue/queues/events-1/

# Check consumer positions
curl -u admin:password \
  http://localhost:9093/api/v1/queues/events/consumers

# If safe, remove old partitions
rm -rf /var/lib/vqueue/queues/events-0/
```

3. **Implement application-level retention**:
```bash
# Cron job to clean old data
0 2 * * * find /var/lib/vqueue/queues -name "*-[0-9]*" -mtime +7 -exec rm -rf {} \;
```

4. **Monitor disk usage**:
```bash
df -h /var/lib/vqueue/queues
```

## Data Integrity Issues

### Issue: CRC mismatch error

**Symptoms**:
```
Error: CRC check failed
```

**Causes**:
- Corrupted message file
- Disk errors
- Incomplete write (system crash)

**Solutions**:

1. **Check disk health**:
```bash
sudo smartctl -a /dev/sda
dmesg | grep -i error
```

2. **Filesystem check**:
```bash
sudo fsck /dev/sda1
```

3. **If queue corrupted**, may need to skip corrupted partition:
```bash
# Backup corrupted files
cp -r /var/lib/vqueue/queues/events-5 /backup/

# Force new partition (advanced - data loss!)
# Remove queue metadata to force new partition
rm /var/lib/vqueue/queues/events_info_queue
```

### Issue: Lost messages

**Symptoms**:
- Messages produced but not consumable
- Missing sequence in offsets

**Causes**:
- Consumer committed without processing
- Application bug
- System crash during write

**Solutions**:

1. **Check queue integrity**:
```bash
# Examine queue files
ls -lh /var/lib/vqueue/queues/my-queue-*/
```

2. **Reset consumer to re-read**:
```bash
rm /var/lib/vqueue/queues/my-queue_info_pop_my-consumer
```

3. **Implement proper commit strategy**:
```python
# Good: Commit after processing
messages = consumer.consume()
for msg in messages:
    process(msg)  # Process first
consumer.commit()  # Then commit

# Bad: Commit before processing
messages = consumer.consume()
consumer.commit()  # Don't commit here!
for msg in messages:
    process(msg)
```

## Configuration Issues

### Issue: Config file not found

**Symptoms**:
```
Warning: Failed to load config file
```

**Causes**:
- Config file missing
- Wrong path
- Wrong filename

**Solutions**:

1. **Create default config**:
```bash
v-queue-server  # Creates default v-queue-server.toml
```

2. **Specify config path**:
```bash
v-queue-server --config /etc/vqueue/server.toml
```

3. **Check current directory**:
```bash
ls -la v-queue-server.toml
pwd
```

### Issue: Config changes not applied

**Symptoms**:
- Changed config but server behavior unchanged

**Causes**:
- Server not restarted
- Wrong config file
- CLI args overriding config

**Solutions**:

1. **Restart server**:
```bash
sudo systemctl restart v-queue-server
```

2. **Verify config file used**:
```bash
v-queue-server --config /path/to/config.toml
```

3. **Check CLI args** (they override config):
```bash
# This overrides config file
v-queue-server --bind 0.0.0.0:9093 --no-auth
```

## Client Issues

### Issue: Empty messages returned

**Symptoms**:
```json
{"messages": []}
```

**Causes**:
- Queue is empty
- Timeout expired with no new messages
- Consumer at end of queue

**Solutions**:

1. **This is normal behavior** - empty queue returns empty array

2. **Use timeout for long polling**:
```python
# Wait up to 30 seconds for messages
messages = consumer.consume("events", "consumer", timeout=30)
```

3. **Check queue status**:
```bash
curl -u admin:password http://localhost:9093/api/v1/queues/events
# Check count_pushed to see if messages exist
```

### Issue: Messages received multiple times

**Symptoms**:
- Same message processed multiple times

**Causes**:
- Not committing after processing
- Consumer crashes before commit
- Multiple consumers with same name

**Solutions**:

1. **Always commit after processing**:
```python
messages = consumer.consume("events", "consumer")
process(messages)
consumer.commit("events", "consumer")  # Must commit!
```

2. **Implement idempotency** in processing logic

3. **Use unique consumer names**:
```python
import uuid
consumer_name = f"my-service-{uuid.uuid4()}"
# Or use hostname
import socket
consumer_name = f"my-service-{socket.gethostname()}"
```

## System Issues

### Issue: Too many open files

**Symptoms**:
```
Error: Too many open files
```

**Causes**:
- OS file descriptor limit reached
- Resource leak
- Many queues/consumers

**Solutions**:

1. **Check current limits**:
```bash
ulimit -n
```

2. **Increase limits**:
```bash
# Temporary
ulimit -n 65536

# Permanent - /etc/security/limits.conf
vqueue soft nofile 65536
vqueue hard nofile 65536
```

3. **Restart server** after changing limits

4. **Check open files**:
```bash
lsof -p $(pgrep v-queue-server) | wc -l
```

### Issue: Permission denied

**Symptoms**:
```
Error: Permission denied
```

**Causes**:
- Wrong user running server
- Incorrect directory permissions
- SELinux/AppArmor restrictions

**Solutions**:

1. **Check permissions**:
```bash
ls -la /var/lib/vqueue/queues
```

2. **Fix ownership**:
```bash
sudo chown -R vqueue:vqueue /var/lib/vqueue
```

3. **Fix permissions**:
```bash
sudo chmod -R 755 /var/lib/vqueue
```

4. **Check SELinux**:
```bash
getenforce  # If enforcing, may be blocking
sudo ausearch -m avc -ts recent  # Check denials
```

## Debugging Tips

### Enable Debug Logging

```bash
v-queue-server --log-level debug
```

Or in config:
```toml
log_level = "debug"
```

### Check Server Logs

```bash
# systemd
journalctl -u v-queue-server -f

# Direct output
v-queue-server 2>&1 | tee server.log
```

### Test with cURL

```bash
# Health check
curl http://localhost:9093/health

# List queues
curl -u admin:password http://localhost:9093/api/v1/queues

# Queue info
curl -u admin:password http://localhost:9093/api/v1/queues/events

# Consume
curl -u admin:password \
  "http://localhost:9093/api/v1/queues/events/consumers/test/messages?timeout_ms=0"
```

### Inspect Queue Files

```bash
# List queues
ls -la /var/lib/vqueue/queues/

# Check queue size
du -sh /var/lib/vqueue/queues/events-0/

# View metadata files (text)
cat /var/lib/vqueue/queues/events_info_queue
cat /var/lib/vqueue/queues/events-0/events_info_push

# Binary message file (don't cat)
ls -lh /var/lib/vqueue/queues/events-0/events_queue
```

### Network Debugging

```bash
# Check port listening
netstat -tlnp | grep 9093

# Monitor connections
netstat -anp | grep 9093

# Packet capture
sudo tcpdump -i any port 9093 -A
```

### Resource Monitoring

```bash
# CPU and memory
top -p $(pgrep v-queue-server)

# Disk I/O
iostat -x 1

# File descriptors
lsof -p $(pgrep v-queue-server)

# System calls
strace -p $(pgrep v-queue-server)
```

## Getting Help

### Collect Diagnostic Information

When reporting issues:

1. **Version information**:
```bash
v-queue-server --version  # Future enhancement
cargo --version
rustc --version
```

2. **Configuration**:
```bash
cat v-queue-server.toml
```

3. **Logs**:
```bash
journalctl -u v-queue-server --since "1 hour ago" > logs.txt
```

4. **System information**:
```bash
uname -a
df -h
free -h
```

5. **Queue state**:
```bash
ls -laR /var/lib/vqueue/queues/ > queue-state.txt
```

### Common Error Messages

| Error | Meaning | Solution |
|-------|---------|----------|
| Connection refused | Server not running | Start server |
| Unauthorized | Auth failed | Check credentials |
| Queue not found | Queue doesn't exist | Create queue by pushing messages via v_queue library or check name |
| Already open | Queue locked | Check for other writers |
| Permission denied | File permissions | Fix ownership/permissions |
| Too many open files | FD limit reached | Increase limit |
| CRC check failed | Data corruption | Check disk health |

## Next Steps

- [Installation Guide](03-installation.md)
- [Configuration Guide](04-configuration.md)
- [Performance Guide](08-performance.md)