vflight

Share files over the Veilid distributed network with content-addressable storage.

Overview

vflight is a Rust library and CLI tool that enables secure, decentralized file sharing using the Veilid network. Files are chunked, hashed with BLAKE3, and stored in a distributed hash table (DHT) for reliable retrieval.

Note: This project provides the core library and CLI interface. GUI applications can be built as separate projects that depend on vflight as a library.

Features

Distributed File Sharing: Share files directly over the Veilid p2p network
Content-Addressable Storage: Files are stored by their cryptographic hash
Chunk-based Transfer: Large files are split into 30KB chunks for efficient transmission
Hash Verification: Each chunk is verified using BLAKE3 hashing
Stream Encryption: ChaCha20-Poly1305 encryption with Argon2id key derivation (--password flag or VFLIGHT_PASSWORD env var)
Private Routes: Uses Veilid's private routes for secure peer communication
Parallel Downloads: Download 8 chunks concurrently by default, configurable via --parallel
Resumable Downloads: Interrupted transfers resume automatically from where they left off
File Compression: Zstd compression with --compress flag reduces transfer size; fetcher decompresses automatically
Bandwidth Throttling: Limit upload and download speed with --throttle <KB/s> (token-bucket algorithm with burst support)
Async/Await: Built on tokio for high-performance async operations
Local Testing Mode: --insecure-local-fallback flag for offline testing in restricted environments (WSL, Docker, etc.)
Performance Metrics: Optional --metrics flag for detailed timing and throughput analysis
Default Logging: Logs at info level by default, customizable via RUST_LOG

Installation

As a Library

Add to your Cargo.toml:

[dependencies]
vflight = "0.9"

As a CLI Tool

cargo install vflight

Quick Start

CLI Usage

Local Testing (Insecure Mode)

For testing in restricted environments (WSL, Docker, or without network access), use the --insecure-local-fallback flag:

# Seed a file locally (stores chunks in /tmp/vflight-local/)
cargo run -- seed --insecure-local-fallback /path/to/file.txt

# Fetch the file back
cargo run -- fetch --insecure-local-fallback file.txt .

Warning: This mode stores chunks as plaintext files without network distribution. Only for development/testing.

Network Mode (Production)

Seed a file (become a provider)

vflight seed /path/to/file.txt

This will:

Start a Veilid node
Split the file into chunks
Store chunks in the DHT
Print the DHT key for retrieval

Fetch a file

vflight fetch <dht-key> /output/dir

This will:

Connect to the Veilid network
Retrieve file metadata
Download all chunks
Verify hashes
Reassemble and save the file

Encrypted Transfers

For sensitive files, use the --password flag (or set the VFLIGHT_PASSWORD environment variable) to encrypt chunks in transit:

# Seed with encryption via flag
vflight seed --password "my-secret-password" /path/to/sensitive-file.txt

# Or via environment variable
export VFLIGHT_PASSWORD="my-secret-password"
vflight seed /path/to/sensitive-file.txt

# Fetch with the same password (flag or env var)
vflight fetch --password "my-secret-password" <dht-key> /output/dir

The --password flag takes precedence over the environment variable when both are set.

Encryption uses:

ChaCha20-Poly1305 for authenticated encryption
Argon2id for password-based key derivation
Unique nonce per chunk to prevent replay attacks

Note: Share the password securely out-of-band. The recipient cannot decrypt without it.

Parallel Downloads

By default, vflight downloads 8 chunks concurrently for faster retrieval. Adjust with --parallel:

# Use 16 parallel downloads for faster transfers
vflight fetch <dht-key> /output/dir --parallel 16

# Use fewer connections on constrained networks
vflight fetch <dht-key> /output/dir --parallel 4

Resumable Downloads

Interrupted downloads resume automatically. If a fetch is interrupted (Ctrl+C, crash, network failure), just run the same command again:

# Start download - gets interrupted
vflight fetch <dht-key> /output/dir
# ^C (interrupted)

# Resume - automatically continues from where it left off
vflight fetch <dht-key> /output/dir
# Output: "Resuming: 50/100 chunks cached"

Resume state is stored in .vflight-resume/ within the output directory. Control resume behavior with flags:

# Start fresh, ignoring any cached chunks
vflight fetch --no-resume <dht-key> /output/dir

# Skip re-verification of cached chunks (faster but less safe)
vflight fetch --trust-cache <dht-key> /output/dir

Note: For encrypted files, the password is only required when there are chunks still to download. If all chunks are cached, no password is needed to resume.

File Compression

Reduce transfer size with zstd compression. Only the --compress flag is needed on seed — the fetcher detects and decompresses automatically:

# Seed with compression
vflight seed --compress /path/to/file.txt

# Fetch (no flag needed — decompresses automatically)
vflight fetch <dht-key> /output/dir

Compression is applied to the entire file before chunking, so hashing, encryption, and resume all work transparently on top of it.

Bandwidth Throttling

Limit transfer speed for both seeding and fetching with --throttle:

# Seed at 500 KB/s upload cap
vflight seed --throttle 500 /path/to/file.txt

# Fetch at 100 KB/s download cap
vflight fetch --throttle 100 <dht-key> /output/dir

The throttler uses a token-bucket algorithm: a burst of up to 10 chunks (300 KB) is allowed immediately, after which throughput is capped at the configured rate. Set --throttle 0 (the default) for unlimited speed.

Library Usage

use vflight::{chunk_file, FileMetadata, CHUNK_SIZE};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Chunk a file
    let chunks = chunk_file(std::path::Path::new("myfile.bin"))?;

    for chunk in chunks {
        println!("Chunk {}: {} bytes, hash: {}",
                 chunk.index,
                 chunk.data.len(),
                 chunk.hash);
    }

    Ok(())
}

Architecture

Modules

chunker: File splitting and reassembly with BLAKE3 hashing
protocol: Message types and wire encoding for network communication
node: Veilid node initialization and lifecycle management
seed: Server-side file seeding logic
fetch: Client-side file retrieval logic
encryption: ChaCha20-Poly1305 encryption with Argon2id key derivation
compression: Zstd file compression before chunking
resume: Resume state management for interrupted downloads
throttle: Token-bucket bandwidth throttling for seed and fetch transfers
metrics: Performance metrics collection and summary reporting

Protocol

The system uses a simple request/response protocol over Veilid AppCall. Requests are JSON-encoded. Responses use a tagged wire format: a single tag byte selects the encoding, followed by the payload.

Requests (JSON):

{"type": "GetMetadata"}
{"type": "GetChunk", "index": 0}

Metadata response (JSON, tag 0x01):

{
  "type": "Metadata",
  "name": "file.txt",
  "size": 102400,
  "total_chunks": 4,
  "chunk_hashes": ["hash1", "hash2", "hash3", "hash4"],
  "compressed": false
}

ChunkData response (binary, tag 0x00):

[1 byte  : 0x00 tag         ]
[8 bytes : chunk index (u64 LE)]
[64 bytes: BLAKE3 hex hash  ]
[N bytes : raw chunk data   ]

ChunkData uses a compact binary layout to avoid the ~33% overhead that base64 + JSON would add. All other response variants (Metadata, Error) are JSON-encoded with tag 0x01.

Configuration

Chunk Size

Default chunk size is 30,000 bytes. To modify, change CHUNK_SIZE in src/protocol.rs.

Namespace

Each node instance can use a different namespace to run multiple instances in the same application:

let (api, rx) = start_node("my_namespace").await?;

Insecure Storage

The secure system keyring is used by default. In environments where the keyring is unavailable (e.g., WSL, Docker), vflight automatically falls back to insecure local storage.

Logging & Debugging

The application logs at info level by default, so you'll see progress messages without extra configuration:

# Default info level (progress messages print automatically)
vflight seed file.txt

# Increase verbosity
RUST_LOG=debug vflight seed file.txt
RUST_LOG=trace vflight seed file.txt

# Disable logging
RUST_LOG=off vflight seed file.txt

# Fine-grained control
RUST_LOG=vflight=debug,veilid_core=info vflight seed file.txt

Log levels:

info: User-visible progress (start, attach, complete)
debug: Operation details (chunk X/Y, DHT operations)
warn: Recoverable errors
error: Critical failures (hash mismatches)
trace: Request/response payloads

Performance Metrics

Use the --metrics flag to print a performance summary after command completion:

# Seed with metrics
vflight seed file.txt --metrics

# Fetch with metrics
vflight fetch <dht-key> ./output --metrics

Example output:

Performance Summary (elapsed: 12.34s)
────────────────────────────────────────────────────────────────────
Category        Count      Total        Avg        Min        Max      Bytes
FileIO              1     0.05ms     0.05ms     0.05ms     0.05ms     1.0 MB
HashCompute        35     0.12ms     0.00ms     0.00ms     0.01ms     1.0 MB
DhtOperation        3     2.15s    716.7ms   210.0ms  1200.0ms         -
ChunkTransfer      35     9.80s    280.0ms   150.0ms   520.0ms     1.0 MB
────────────────────────────────────────────────────────────────────

Metrics categories:

FileIO: File read/write operations
HashCompute: BLAKE3 hash computation
DhtOperation: DHT create, open, read, write operations
ChunkTransfer: Network chunk transfer latency

Testing

Run the comprehensive test suite:

cargo test

Tests include:

File chunking and reassembly
Hash consistency and verification
Protocol serialization/deserialization
Encryption/decryption roundtrips
File I/O operations
CLI argument parsing

Performance

Chunk Size: 30 KB per chunk (configurable)
Hash Algorithm: BLAKE3 (extremely fast, cryptographically secure)
Network: Uses Veilid's optimized routing

Security Considerations

Files are stored by content hash, providing deduplication and integrity
Veilid handles encryption and authentication at the network layer
Private routes are used to prevent direct peer identification
Optional stream encryption (ChaCha20-Poly1305) for end-to-end confidentiality
Password-based keys derived with Argon2id (memory-hard, resistant to GPU attacks)
Always verify file hashes after download

Limitations

Currently requires Veilid network access
File metadata must fit in a single DHT record
Chunk order must be preserved during retrieval

Contributing

Contributions are welcome! Please:

Write tests for new features
Ensure all tests pass: cargo test
Follow Rust conventions and idioms
Update documentation

License

MIT License - see LICENSE file for details

Support

For issues, questions, or contributions:

GitHub Issues: danzbyrd/vflight
Veilid Community: veilid.com

Roadmap

Phase 1: Logging & Observability

Add #[instrument] macros to async functions for distributed tracing
Structured error logging with context information
Request/response logging for debugging network issues
Performance metrics collection

Phase 2: Core Features

Stream encryption for chunks (in-flight security)
Parallel chunk downloads for faster retrieval
Resume interrupted transfers
File compression support
Bandwidth throttling controls

Phase 3: Advanced Features

Directory/recursive file support
S3-compatible API endpoint
Peer discovery improvements
DHT replication strategies

vflight 0.9.1