# Filesystem Interface Implementation Summary
This document describes the architecture and implementation details of the Heroforge filesystem interface (`fs_interface`), a high-level abstraction that provides filesystem-like operations over a SQLite-backed version control repository.
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ FsInterface (sync API) │
│ - RwLock<StagingState> │
│ - Author name (set at initialization) │
│ - All read/write operations acquire appropriate locks │
└─────────────────────────────────────────────────────────────────┘
│
┌───────────────────┴───────────────────┐
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────────┐
│ Staging Directory │ │ Commit Thread │
│ │ │ (background) │
│ - All writes go │ │ │
│ here first │ │ - Runs every 1 minute │
│ - Files < 2MB │ │ - Acquires write lock │
│ - Frequent updates │ │ - Blocks all I/O │
│ allowed │ │ - Flushes to .forge DB │
│ │ │ - Clears staging dir │
└─────────────────────┘ └─────────────────────────┘
│ │
└───────────────────┬───────────────────┘
│
▼
┌─────────────────────┐
│ .forge Database │
│ (SQLite) │
│ │
│ - Committed files │
│ - Version history │
│ - Manifests │
└─────────────────────┘
```
## Core Concepts
### Staging Directory
All write operations go to a **staging directory** first, not directly to the SQLite database. This provides:
1. **Fast writes**: Writing to the filesystem is faster than SQLite transactions
2. **Frequent updates**: Files can be modified many times before commit
3. **Atomic commits**: All staged changes are committed together
4. **Crash recovery**: Uncommitted work is recoverable from the staging area
**Current Limitations:**
- Files larger than **2 MB** are not supported
- Staging directory is local to the repository
### Read Path (Layered Lookup)
When reading a file, the interface checks locations in this order:
```
1. Staging Directory → If file exists here, return it (most recent)
│
▼ (not found)
2. .forge Database → Query SQLite for committed version
│
▼ (not found)
3. Return Error → File does not exist
```
This ensures reads always see the most recent version, whether committed or staged.
### Write Path
All writes follow this flow:
```
1. Validate path and content size (< 2MB)
2. Acquire write lock (RwLock)
3. Write file to staging directory
4. Update staging state metadata
5. Release lock
6. Return success (file is NOT yet in .forge DB)
```
The actual commit to SQLite happens asynchronously via the commit thread.
### Partial File Updates (Read-Modify-Write)
When you need to modify only part of a file (e.g., update a few bytes in the middle), the interface uses a **read-modify-write** pattern. The file is promoted to staging if not already there:
```
Scenario: Modify bytes 100-200 in "data.bin" (file exists in .forge DB)
┌─────────────────────────────────────────────────────────────────┐
│ Step 1: Check staging directory │
│ → File NOT in staging │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 2: Read full file from .forge DB │
│ → Load entire "data.bin" into memory │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 3: Apply modification in memory │
│ → Overwrite bytes 100-200 with new content │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 4: Write complete file to staging directory │
│ → "data.bin" now exists in staging (full copy) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 5: Subsequent reads/writes use staging copy │
│ → .forge DB version is now shadowed │
└─────────────────────────────────────────────────────────────────┘
```
**Key behaviors:**
1. **Promotion to staging**: The first write to a committed file copies it entirely to staging
2. **Subsequent writes are fast**: Once in staging, modifications happen directly on the staged file
3. **No delta storage in staging**: Staging always holds complete files, not patches
4. **Memory efficient**: Only one file loaded at a time during modification
```rust
// Example: Multiple partial updates to the same file
let fs = FsInterface::new(repo, "author")?;
// First write: file promoted from .forge DB → staging
fs.write_at("config.bin", 0, b"HEADER")?; // Reads from DB, writes to staging
// Second write: file already in staging, fast update
fs.write_at("config.bin", 100, b"DATA")?; // Modifies staging directly
// Third write: still in staging
fs.write_at("config.bin", 200, b"FOOTER")?; // Modifies staging directly
// All three changes committed together at next auto-commit
```
**Flow diagram for file already in staging:**
```
Scenario: Modify bytes 100-200 in "data.bin" (file already in staging)
┌─────────────────────────────────────────────────────────────────┐
│ Step 1: Check staging directory │
│ → File IS in staging │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 2: Read file from staging (or just seek to offset) │
│ → Fast local file access │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 3: Apply modification directly │
│ → Overwrite bytes 100-200 in staging file │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Step 4: Done - no DB access needed │
│ → Very fast for repeated modifications │
└─────────────────────────────────────────────────────────────────┘
```
This design optimizes for the common case of **frequent small updates** to the same files, which is typical for:
- Configuration files being edited
- Log files being appended
- Database files with incremental changes
- Cache files being updated
## Threading Model: Sync + Threads
We use **synchronous I/O with background threads** rather than async/await. This design choice was made because:
| Mental model | Simpler, blocking operations | Complex, non-blocking |
| SQLite compatibility | Excellent (single-writer fits naturally) | Overhead from async wrappers |
| 1-min auto-commit | Natural "stop the world" with RwLock | Tricky lock coordination |
| Error handling | Straightforward | Complex across await points |
| File I/O | Inherently blocking anyway | Just moves to thread pool |
### Thread Responsibilities
```
┌────────────────────────────────────────────────────────────────┐
│ Main Thread(s) │
│ │
│ - Handle user API calls (read_file, write_file, etc.) │
│ - Acquire RwLock for read (shared) or write (exclusive) │
│ - Perform staging directory operations │
│ - Return immediately after staging (writes don't wait) │
└────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ Commit Thread │
│ (background) │
│ │
│ - Spawned at FsInterface initialization │
│ - Sleeps for 1 minute intervals │
│ - On wake: acquires EXCLUSIVE write lock │
│ - While holding lock: │
│ 1. Block ALL reads and writes │
│ 2. Collect all files from staging directory │
│ 3. Create new commit in .forge database │
│ 4. Clear staging directory │
│ 5. Update internal state │
│ - Release lock, go back to sleep │
└────────────────────────────────────────────────────────────────┘
```
## Auto-Commit Behavior
### Timing
- Commits happen automatically every **1 minute**
- This interval balances:
- Responsiveness (changes are persisted relatively quickly)
- Performance (not committing after every tiny change)
- Atomicity (groups related changes together)
### Blocking During Commit
When the commit thread runs:
1. It acquires an **exclusive write lock**
2. **ALL operations are blocked** (reads AND writes)
3. This is intentional - ensures consistent state
4. Typical commit duration: milliseconds to a few seconds
5. Lock is released immediately after commit
```
Timeline:
─────────────────────────────────────────────────────────►
│ │
▼ ▼
[Normal Operations] [Commit in Progress]
read ✓ read ✗ (blocked)
write ✓ write ✗ (blocked)
│ │
▼ ▼
────────────────────────────────────────────────────────►
[Commit Complete]
read ✓
write ✓
```
### Forced Commits
Certain operations force an **immediate commit** regardless of the timer:
- **Branch change**: Must commit current work before switching
- **Tag creation**: Tags reference specific commits
- **Checkout**: Switching to a different state
- **Repository close**: Graceful shutdown
```rust
// Pseudocode for forced commit triggers
fn change_branch(&self, branch: &str) -> Result<()> {
self.force_commit()?; // Commit staging before branch change
self.switch_branch_internal(branch)?;
Ok(())
}
```
## Author Identification
The `FsInterface` requires an **author name** at initialization time. This author is used for all commits created by this interface instance.
```rust
// Initialize with author
let fs = FsInterface::new(repo, "developer@example.com")?;
// All commits from this instance will be attributed to "developer@example.com"
fs.write_file("config.json", b"{}")?; // Author: developer@example.com
```
### Rationale
- Simplifies the API (no author parameter on every write)
- Ensures consistent attribution
- Maps well to application users/services
- Different authors can use different `FsInterface` instances
## File Size Limits
**Current limit: 2 MB per file**
Files exceeding this limit are rejected with an error. This limitation exists because:
1. SQLite BLOB handling for large files is inefficient
2. Staging directory would grow too large
3. Memory pressure during commit operations
4. Future: Large file support will use chunked storage
```rust
// This will fail
fs.write_file("large_video.mp4", &large_content)?; // Error: File too large
// Workaround: Split into chunks (future API)
// fs.write_large_file("video.mp4", stream)?;
```
## Locking Strategy
### RwLock Design
```rust
struct StagingState {
files: HashMap<PathBuf, StagedFile>,
dirty: bool,
last_commit: Instant,
}
struct FsInterface {
staging: RwLock<StagingState>,
// ...
}
```
### Lock Acquisition
| `exists()` | Read | Nothing |
| `read_file()` | Read | Nothing |
| `list_dir()` | Read | Nothing |
| `stat()` | Read | Nothing |
| `write_file()` | Write | Other writes |
| `delete_file()` | Write | Other writes |
| `move_file()` | Write | Other writes |
| **Commit** | **Exclusive** | **Everything** |
### Deadlock Prevention
- Single lock for all staging state
- No nested lock acquisition
- Commit thread uses try_lock with timeout
- Operations are short-lived
## Error Handling
```rust
pub enum FsError {
// Path errors
InvalidPath(String),
NotFound(String),
NotAFile(String),
NotADirectory(String),
// Size errors
FileTooLarge { path: String, size: u64, max: u64 },
// Lock errors
LockTimeout,
CommitInProgress,
// Storage errors
DatabaseError(String),
StagingError(String),
// Other
Encoding(String),
TransactionError(String),
}
```
## Proposed Repository Reorganization
The current repository structure has grown organically and needs reorganization around functional boundaries.
### Current Structure (flat)
```
src/
├── artifact/ # Blob and manifest handling
├── error.rs # Global errors
├── examples/ # Example code
├── fs/ # Filesystem interface (NEW)
├── hash.rs # Hashing utilities
├── lib.rs # Main entry point
├── repo/ # Repository + builders
├── server/ # QUIC server
└── sync/ # Sync protocol
```
### Proposed Structure (modular)
```
src/
├── lib.rs # Minimal re-exports only
│
├── core/ # Core types and utilities
│ ├── mod.rs
│ ├── error.rs # Unified error types
│ ├── hash.rs # Hashing (BLAKE3, SHA3)
│ └── README.md
│
├── db/ # Database layer
│ ├── mod.rs
│ ├── sqlite.rs # SQLite operations
│ ├── schema.rs # Table definitions
│ ├── migrations.rs # Schema migrations
│ └── README.md
│
├── artifact/ # Content-addressable storage
│ ├── mod.rs
│ ├── blob.rs # Raw blob storage
│ ├── delta.rs # Delta compression
│ ├── manifest.rs # Directory manifests
│ └── README.md
│
├── fs_interface/ # External filesystem API
│ ├── mod.rs
│ ├── staging.rs # Staging directory management
│ ├── commit_thread.rs # Background commit worker
│ ├── operations.rs # Read/write operations
│ ├── transaction.rs # Transaction handling
│ ├── errors.rs # FS-specific errors
│ └── README.md
│
├── repo/ # Repository management
│ ├── mod.rs
│ ├── repository.rs # Main Repository type
│ ├── branches.rs # Branch operations
│ ├── tags.rs # Tag operations
│ ├── history.rs # Commit history
│ ├── import/ # Import from other VCS
│ │ ├── mod.rs
│ │ └── git.rs
│ └── README.md
│
├── network/ # Network layer
│ ├── mod.rs
│ ├── protocol.rs # Wire protocol definitions
│ ├── quic/ # QUIC transport
│ │ ├── mod.rs
│ │ ├── client.rs
│ │ └── server.rs
│ └── README.md
│
├── sync/ # Synchronization logic
│ ├── mod.rs
│ ├── push.rs # Push operations
│ ├── pull.rs # Pull operations
│ ├── merge.rs # Merge strategies
│ └── README.md
│
└── examples/ # Example applications
├── basic_usage.rs
├── fs_operations.rs
├── sync_demo.rs
└── server_demo.rs
```
### Module Responsibilities
| `core` | Shared types, error handling, hashing |
| `db` | All SQLite interaction, schema, migrations |
| `artifact` | Content storage, compression, deduplication |
| `fs_interface` | External API for filesystem-like access |
| `repo` | High-level repository operations |
| `network` | Transport protocols (QUIC, future HTTP) |
| `sync` | Push/pull/merge logic |
| `examples` | Runnable example code |
### README in Each Module
Each module should have a `README.md` containing:
1. **Purpose**: What this module does
2. **Key Types**: Main structs/traits/enums
3. **Dependencies**: What other modules it uses
4. **Thread Safety**: Concurrency notes
5. **Examples**: Code snippets
Example `fs_interface/README.md`:
```markdown
# fs_interface
Provides a filesystem-like API over Heroforge repositories.
## Key Types
- `FsInterface`: Main entry point, created from Repository
- `StagingState`: Tracks uncommitted changes
- `CommitWorker`: Background thread for auto-commits
## Dependencies
- `core`: Error types, hashing
- `db`: SQLite operations
- `artifact`: Blob storage
## Thread Safety
Uses `RwLock<StagingState>` for concurrent access.
Background commit thread acquires exclusive lock every 1 minute.
## Example
\`\`\`rust
let fs = repo.fs_interface("author@example.com")?;
fs.write_file("hello.txt", b"world")?;
let content = fs.read_file("hello.txt")?;
\`\`\`
```
## Future Enhancements
1. **Large file support**: Chunked storage for files > 2MB
2. **Configurable commit interval**: Allow users to set auto-commit timing
3. **Watch API**: Notify on file changes
4. **Conflict resolution**: Handle concurrent modifications
5. **Partial checkouts**: Only stage subset of repository
## See Also
- `src/fs_interface/README.md` - Detailed module documentation
- `examples/fs_operations.rs` - Working example code
- `docs/ARCHITECTURE.md` - Overall system design