# vflight
Share files over the Veilid distributed network with content-addressable storage.
## Overview
`vflight` is a Rust library and CLI tool that enables secure, decentralized file sharing using the [Veilid](https://veilid.com/) network. Files are chunked, hashed with BLAKE3, and stored in a distributed hash table (DHT) for reliable retrieval.
**Note**: This project provides the core library and CLI interface. GUI applications can be built as separate projects that depend on `vflight` as a library.
## Features
- **Distributed File Sharing**: Share files directly over the Veilid p2p network
- **Content-Addressable Storage**: Files are stored by their cryptographic hash
- **Chunk-based Transfer**: Large files are split into 30KB chunks for efficient transmission
- **Hash Verification**: Each chunk is verified using BLAKE3 hashing
- **Stream Encryption**: ChaCha20-Poly1305 encryption with Argon2id key derivation (`--password` flag or `VFLIGHT_PASSWORD` env var)
- **Private Routes**: Uses Veilid's private routes for secure peer communication
- **Parallel Downloads**: Download 8 chunks concurrently by default, configurable via `--parallel`
- **Resumable Downloads**: Interrupted transfers resume automatically from where they left off
- **File Compression**: Zstd compression with `--compress` flag reduces transfer size; fetcher decompresses automatically
- **Bandwidth Throttling**: Limit upload and download speed with `--throttle <KB/s>` (token-bucket algorithm with burst support)
- **Async/Await**: Built on tokio for high-performance async operations
- **Local Testing Mode**: `--insecure-local-fallback` flag for offline testing in restricted environments (WSL, Docker, etc.)
- **Performance Metrics**: Optional `--metrics` flag for detailed timing and throughput analysis
- **Default Logging**: Logs at info level by default, customizable via RUST_LOG
## Installation
### As a Library
Add to your `Cargo.toml`:
```toml
[dependencies]
vflight = "0.9.2"
```
### As a CLI Tool
```bash
cargo install vflight
```
## Quick Start
### CLI Usage
#### Local Testing (Insecure Mode)
For testing in restricted environments (WSL, Docker, or without network access), use the `--insecure-local-fallback` flag:
```bash
# Seed a file locally (stores chunks in /tmp/vflight-local/)
cargo run -- seed --insecure-local-fallback /path/to/file.txt
# Fetch the file back
cargo run -- fetch --insecure-local-fallback file.txt .
```
**Warning**: This mode stores chunks as plaintext files without network distribution. Only for development/testing.
#### Network Mode (Production)
##### Seed a file (become a provider)
```bash
vflight seed /path/to/file.txt
```
This will:
1. Start a Veilid node
2. Split the file into chunks
3. Store chunks in the DHT
4. Print the DHT key for retrieval
#### Fetch a file
```bash
vflight fetch <dht-key> /output/dir
```
This will:
1. Connect to the Veilid network
2. Retrieve file metadata
3. Download all chunks
4. Verify hashes
5. Reassemble and save the file
#### Encrypted Transfers
For sensitive files, use the `--password` flag (or set the `VFLIGHT_PASSWORD` environment variable) to encrypt chunks in transit:
```bash
# Seed with encryption via flag
vflight seed --password "my-secret-password" /path/to/sensitive-file.txt
# Or via environment variable
export VFLIGHT_PASSWORD="my-secret-password"
vflight seed /path/to/sensitive-file.txt
# Fetch with the same password (flag or env var)
vflight fetch --password "my-secret-password" <dht-key> /output/dir
```
The `--password` flag takes precedence over the environment variable when both are set.
Encryption uses:
- **ChaCha20-Poly1305** for authenticated encryption
- **Argon2id** for password-based key derivation
- Unique nonce per chunk to prevent replay attacks
**Note**: Share the password securely out-of-band. The recipient cannot decrypt without it.
#### Parallel Downloads
By default, vflight downloads 8 chunks concurrently for faster retrieval. Adjust with `--parallel`:
```bash
# Use 16 parallel downloads for faster transfers
vflight fetch <dht-key> /output/dir --parallel 16
# Use fewer connections on constrained networks
vflight fetch <dht-key> /output/dir --parallel 4
```
#### Resumable Downloads
Interrupted downloads resume automatically. If a fetch is interrupted (Ctrl+C, crash, network failure), just run the same command again:
```bash
# Start download - gets interrupted
vflight fetch <dht-key> /output/dir
# ^C (interrupted)
# Resume - automatically continues from where it left off
vflight fetch <dht-key> /output/dir
# Output: "Resuming: 50/100 chunks cached"
```
Resume state is stored in `.vflight-resume/` within the output directory. Control resume behavior with flags:
```bash
# Start fresh, ignoring any cached chunks
vflight fetch --no-resume <dht-key> /output/dir
# Skip re-verification of cached chunks (faster but less safe)
vflight fetch --trust-cache <dht-key> /output/dir
```
**Note**: For encrypted files, the password is only required when there are chunks still to download. If all chunks are cached, no password is needed to resume.
#### File Compression
Reduce transfer size with zstd compression. Only the `--compress` flag is needed on seed — the fetcher detects and decompresses automatically:
```bash
# Seed with compression
vflight seed --compress /path/to/file.txt
# Fetch (no flag needed — decompresses automatically)
vflight fetch <dht-key> /output/dir
```
Compression is applied to the entire file before chunking, so hashing, encryption, and resume all work transparently on top of it.
#### Bandwidth Throttling
Limit transfer speed for both seeding and fetching with `--throttle`:
```bash
# Seed at 500 KB/s upload cap
vflight seed --throttle 500 /path/to/file.txt
# Fetch at 100 KB/s download cap
vflight fetch --throttle 100 <dht-key> /output/dir
```
The throttler uses a token-bucket algorithm: a burst of up to 10 chunks (300 KB) is allowed immediately, after which throughput is capped at the configured rate. Set `--throttle 0` (the default) for unlimited speed.
### Library Usage
```rust
use vflight::{chunk_file, FileMetadata, CHUNK_SIZE};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Chunk a file
let chunks = chunk_file(std::path::Path::new("myfile.bin"))?;
for chunk in chunks {
println!("Chunk {}: {} bytes, hash: {}",
chunk.index,
chunk.data.len(),
chunk.hash);
}
Ok(())
}
```
## Architecture
### Modules
- **`chunker`**: File splitting and reassembly with BLAKE3 hashing
- **`protocol`**: Message types and wire encoding for network communication
- **`node`**: Veilid node initialization and lifecycle management
- **`seed`**: Server-side file seeding logic
- **`fetch`**: Client-side file retrieval logic
- **`encryption`**: ChaCha20-Poly1305 encryption with Argon2id key derivation
- **`compression`**: Zstd file compression before chunking
- **`resume`**: Resume state management for interrupted downloads
- **`throttle`**: Token-bucket bandwidth throttling for seed and fetch transfers
- **`metrics`**: Performance metrics collection and summary reporting
### Protocol
The system uses a simple request/response protocol over Veilid AppCall. Requests are JSON-encoded. Responses use a tagged wire format: a single tag byte selects the encoding, followed by the payload.
**Requests (JSON):**
```json
{"type": "GetMetadata"}
{"type": "GetChunk", "index": 0}
```
**Metadata response (JSON, tag `0x01`):**
```json
{
"type": "Metadata",
"name": "file.txt",
"size": 102400,
"total_chunks": 4,
"chunk_hashes": ["hash1", "hash2", "hash3", "hash4"],
"compressed": false
}
```
**ChunkData response (binary, tag `0x00`):**
```text
[1 byte : 0x00 tag ]
[8 bytes : chunk index (u64 LE)]
[64 bytes: BLAKE3 hex hash ]
[N bytes : raw chunk data ]
```
ChunkData uses a compact binary layout to avoid the ~33% overhead that base64 + JSON would add. All other response variants (`Metadata`, `Error`) are JSON-encoded with tag `0x01`.
## Configuration
### Chunk Size
Default chunk size is 30,000 bytes. To modify, change `CHUNK_SIZE` in `src/protocol.rs`.
### Namespace
Each node instance can use a different namespace to run multiple instances in the same application:
```rust
let (api, rx) = start_node("my_namespace").await?;
```
### Insecure Storage
The secure system keyring is used by default. In environments where the keyring is unavailable (e.g., WSL, Docker), vflight automatically falls back to insecure local storage.
## Logging & Debugging
The application logs at **info** level by default, so you'll see progress messages without extra configuration:
```bash
# Default info level (progress messages print automatically)
vflight seed file.txt
# Increase verbosity
RUST_LOG=debug vflight seed file.txt
RUST_LOG=trace vflight seed file.txt
# Disable logging
RUST_LOG=off vflight seed file.txt
# Fine-grained control
RUST_LOG=vflight=debug,veilid_core=info vflight seed file.txt
```
Log levels:
- **info**: User-visible progress (start, attach, complete)
- **debug**: Operation details (chunk X/Y, DHT operations)
- **warn**: Recoverable errors
- **error**: Critical failures (hash mismatches)
- **trace**: Request/response payloads
## Performance Metrics
Use the `--metrics` flag to print a performance summary after command completion:
```bash
# Seed with metrics
vflight seed file.txt --metrics
# Fetch with metrics
vflight fetch <dht-key> ./output --metrics
```
Example output:
```
Performance Summary (elapsed: 12.34s)
────────────────────────────────────────────────────────────────────
Category Count Total Avg Min Max Bytes
FileIO 1 0.05ms 0.05ms 0.05ms 0.05ms 1.0 MB
HashCompute 35 0.12ms 0.00ms 0.00ms 0.01ms 1.0 MB
DhtOperation 3 2.15s 716.7ms 210.0ms 1200.0ms -
ChunkTransfer 35 9.80s 280.0ms 150.0ms 520.0ms 1.0 MB
────────────────────────────────────────────────────────────────────
```
Metrics categories:
- **FileIO**: File read/write operations
- **HashCompute**: BLAKE3 hash computation
- **DhtOperation**: DHT create, open, read, write operations
- **ChunkTransfer**: Network chunk transfer latency
## Testing
Run the comprehensive test suite:
```bash
cargo test
```
Tests include:
- File chunking and reassembly
- Hash consistency and verification
- Protocol serialization/deserialization
- Encryption/decryption roundtrips
- File I/O operations
- CLI argument parsing
## Performance
- **Chunk Size**: 30 KB per chunk (configurable)
- **Hash Algorithm**: BLAKE3 (extremely fast, cryptographically secure)
- **Network**: Uses Veilid's optimized routing
## Security Considerations
- Files are stored by content hash, providing deduplication and integrity
- Veilid handles encryption and authentication at the network layer
- Private routes are used to prevent direct peer identification
- Optional stream encryption (ChaCha20-Poly1305) for end-to-end confidentiality
- Password-based keys derived with Argon2id (memory-hard, resistant to GPU attacks)
- Always verify file hashes after download
## Limitations
- Currently requires Veilid network access
- File metadata must fit in a single DHT record
- Chunk order must be preserved during retrieval
## Contributing
Contributions are welcome! Please:
1. Write tests for new features
2. Ensure all tests pass: `cargo test`
3. Follow Rust conventions and idioms
4. Update documentation
## License
MIT License - see LICENSE file for details
## Support
For issues, questions, or contributions:
- GitHub Issues: [danzbyrd/vflight](https://github.com/danzbyrd/vflight/issues)
- Veilid Community: [veilid.com](https://veilid.com/)
## Roadmap
### Phase 1: Logging & Observability
- [x] Add `#[instrument]` macros to async functions for distributed tracing
- [x] Structured error logging with context information
- [x] Request/response logging for debugging network issues
- [x] Performance metrics collection
### Phase 2: Core Features
- [x] Stream encryption for chunks (in-flight security)
- [x] Parallel chunk downloads for faster retrieval
- [x] Resume interrupted transfers
- [x] File compression support
- [x] Bandwidth throttling controls
### Phase 3: Advanced Features
- [ ] Directory/recursive file support
- [ ] S3-compatible API endpoint
- [ ] Peer discovery improvements
- [ ] DHT replication strategies