dx-forge 0.0.1

Ultra-fast file watcher library with dual-mode event system (rapid <35µs, quality <60µs) for DX tools
Documentation

Forge - Production-Ready File Watcher Library# Forge - Ultra-Fast CRDT Version Control

Ultra-fast file watcher library with dual-mode event system optimized for DX tools.Operation-level version control powered by CRDTs. Forge tracks file edits as fine-grained operations, persists them in a local DeltaDB, and keeps peers in sync over WebSockets.

✨ Features## ⚡⚡ Dual-Watcher Architecture

  • ⚡ Rapid Events: <35µs ultra-fast change notifications (typically 1-2µs)Forge uses a dual-watcher system for maximum performance and quality:

  • 📊 Quality Events: <60µs full operation detection with line numbers and diffs

  • 🚀 Production Ready: Zero environment variables, optimal hardcoded settings### 🚀 Mode 1: RAPID Detection (<20µs)

  • 🔧 CRDT-based: Conflict-free replicated data types for distributed sync

  • 💾 Memory-mapped I/O: Leverages OS page cache for sub-microsecond reads- Zero syscalls - Uses atomic sequence counter (no time calls!)

  • 🎯 DX-focused: Built specifically as a base for developer experience tools- No file operations - Skips metadata, mtime, and content reads

  • Instant feedback - Ultra-fast change logging

🚀 Quick Start- Target: <20µs - Ultra-fast notification system

As a Library Dependency### 📊 Mode 2: QUALITY Detection (<60µs)

Add to your Cargo.toml:- Full file analysis - Complete operation detection with line numbers

  • Rich metadata - Diffs, timestamps, and sync details

[dependencies]- **Target: <60µs** - Fast detailed analysis

forge = "1.0"

tokio = { version = "1.48", features = ["full"] }Both modes run sequentially for every file change, providing instant feedback (rapid) followed by complete details (quality).

🎯 Performance Targets

Basic Usage

  • RAPID mode: <20µs change detection ✅ ACHIEVED: 3-20µs

use forge::{ForgeWatcher, ForgeEvent};- **Total latency**: <320µs for complete processing

- **Debounce**: 1ms ultra-fast mode

#[tokio::main]- **Inspired by**: dx-style project's <100µs techniques

async fn main() -> anyhow::Result<()> {

    // Create watcher for current directory### Current Performance

    let watcher = ForgeWatcher::new(".", false, vec![]).await?;


```bash

// Run the watcher# Small appends (cached, best case)

watcher.run().await?;⚡ [RAPID 3µs] test.txt changed

✨ [QUALITY 58µs | total 61µs] test.txt - 1 ops

Ok(())

}# Regular edits (typical case)


🐢 [QUALITY 301µs | total 321µs] test.txt - 1 ops

### Running Examples```



```bash**RAPID mode**: ✅ Target exceeded (3µs is 6x faster than 20µs goal!)

## Simple watcher example**QUALITY mode**: ⚠️ 58-301µs (varies by edit type - appends are fast, full diffs slower)

cargo run --release --example simple

## Quick Start

## Full CLI with all features

cargo run --release --bin forge```bash

```# Default mode (dual-watcher enabled)

cargo run --release

## 🎯 Dual-Event System

# Enable profiling to see timings

Forge emits **two types of events** for every file change:DX_WATCH_PROFILE=1 cargo run --release



### 1. ⚡ Rapid Event (<35µs)# Disable rapid mode (quality only, for testing)

DX_DISABLE_RAPID_MODE=1 cargo run --release

Ultra-fast notification using zero syscalls:

- **Timing**: Typically 1-2µs, max 35µs# Example output:

- **Purpose**: Instant UI feedback for formatters/linters# ⚡ [RAPID 8µs] README.md changed

- **Method**: Atomic sequence counter (no file I/O)# ✨ [QUALITY 52µs | total 60µs] README.md - 1 ops

- **Data**: File path + timing only```



### 2. ✨ Quality Event (<60µs)  ## Configuration



Complete operation detection with details:### Environment Variables

- **Timing**: Typically <60µs

- **Purpose**: Full analysis for quality tools- `DX_WATCH_PROFILE=1` - Show detailed timing for both modes

- **Method**: Memory-mapped I/O + SIMD diffs- `DX_DISABLE_RAPID_MODE=1` - Disable rapid mode (quality only)

- **Data**: Operations, line numbers, content changes- `DX_DEBOUNCE_MS=1` - Debounce interval (default: 1ms)



## 📊 Performance Benchmarks### Performance Markers



```- ⚡ RAPID mode ≤20µs (target achieved)

Rapid Mode (Change Detection):- 🐌 RAPID mode >20µs (needs optimization)

  ⚡ Best case:  1-2µs  (cached, atomic only)- ✨ QUALITY mode ≤60µs (target achieved)  

  ⚡ Typical:    8-20µs (95th percentile)- 🐢 QUALITY mode >60µs (needs optimization)

  🎯 Target:    <35µs  ✅ ACHIEVED

**Clean output - only shows when there are changes!**

Quality Mode (Full Analysis):

  ✨ Best case:  58µs   (simple append)Testing no-op detection...

  ✨ Typical:    60µs   (typical edits)
  🐢 Worst case: 301µs  (complex diffs)
  🎯 Target:    <60µs  ⚠️ MOSTLY ACHIEVED

Example Output

⚡ [RAPID 8µs] test.txt changed
✨ [QUALITY 52µs | total 60µs]

- test.txt @ 1:1
    Hello, Forge!

🏗️ Architecture

Core Components

  • Watcher: File system monitoring with 1ms debounce
  • CRDT: Automerge + Yrs for conflict-free merging
  • Storage: SQLite + LZ4 compression for operation log
  • Sync: WebSocket-based real-time synchronization
  • Cache: Memory-mapped file pool + OS page cache warming

Event Flow

File Change
    ↓
[1ms Debounce]
    ↓
⚡ Rapid Detection (1-2µs)
    ↓ atomic counter
    ↓ emit rapid event
    ↓
✨ Quality Detection (60µs)
    ↓ mmap read
    ↓ SIMD diff
    ↓ emit quality event

🔧 Configuration

Production Settings (Hardcoded)

const RAPID_MODE_ENABLED: bool = true;   // Always on
const DEBOUNCE_MS: u64 = 1;              // 1ms optimal
const RAPID_TARGET_US: u128 = 35;        // <35µs rapid
const QUALITY_TARGET_US: u128 = 60;      // <60µs quality

No environment variables needed - everything is optimized out of the box!

📦 Use Cases

Perfect for building DX tools that need:

  1. Formatters: Rapid events trigger instant formatting
  2. Linters: Quality events provide detailed analysis
  3. Hot Reload: Rapid events for instant dev server refresh
  4. Build Tools: Quality events for incremental compilation
  5. Test Runners: Rapid events trigger test re-runs

🎨 Example: Custom Event Handler

use forge::{ForgeWatcher, Operation, OperationType};

## [tokio::main]

async fn main() -> anyhow::Result<()> {
    let watcher = ForgeWatcher::new("./src", false, vec![]).await?;

    // The watcher automatically handles events internally
    // and prints detailed logs for rapid + quality events

    watcher.run().await?;
    Ok(())
}

🚀 Advanced Features

CRDT Synchronization

use forge::ForgeWatcher;

## [tokio::main]

async fn main() -> anyhow::Result<()> {
    // Enable sync with remote peer
    let watcher = ForgeWatcher::new(
        ".",
        true,  // enable_sync
        vec!["ws://localhost:3000/ws".to_string()]
    ).await?;

    watcher.run().await?;
    Ok(())
}

Operation Log Access

use forge::{Database, OperationLog};
use std::sync::Arc;

let db = Database::new(".dx/forge")?;
let oplog = Arc::new(OperationLog::new(Arc::new(db)));

// Query operations for a file
// oplog.get_file_operations("src/main.rs")?;

🎯 Performance Optimization Techniques

  1. Zero-copy I/O: Memory-mapped files via memmap2
  2. SIMD acceleration: Fast byte comparison via memchr
  3. Atomic operations: Lock-free sequence counters
  4. OS cache warming: Pre-load all files at startup
  5. Lazy allocation: Defer expensive operations until needed
  6. Parallel processing: rayon for large file diffs

📝 License

MIT OR Apache-2.0

🙏 Credits

Performance techniques inspired by:

🔗 Links

Built with ❤️ for the DX community