Module incremental 

Incremental Update System

This module implements Thread’s incremental update system for dependency-aware invalidation and targeted re-analysis. It adapts patterns from ReCoco’s FieldDefFingerprint design to Thread’s AST analysis domain.

Architecture

The system consists of four integrated subsystems:

• Types (types): Core data structures for fingerprints, dependency edges, and the dependency graph.
• Graph (graph): Dependency graph traversal algorithms including BFS affected-file detection, topological sort, and cycle detection.
• Storage (storage): Trait definitions for persisting dependency graphs and fingerprints across sessions.
• Backends (backends): Concrete storage implementations (Postgres, D1, InMemory) with factory pattern for runtime backend selection.

Design Pattern

Adapted from ReCoco’s FieldDefFingerprint (analyzer.rs:69-84):

• Source tracking: Identifies which files contribute to each analysis result
• Fingerprint composition: Detects content AND logic changes via Blake3 hashing
• Dependency graph: Maintains import/export relationships for cascading invalidation

Examples

Basic Dependency Graph Operations

use thread_flow::incremental::types::{
    AnalysisDefFingerprint, DependencyEdge, DependencyType,
};
use thread_flow::incremental::graph::DependencyGraph;
use std::path::PathBuf;
use std::collections::HashSet;

// Create a dependency graph
let mut graph = DependencyGraph::new();

// Add a dependency edge: main.rs imports utils.rs
graph.add_edge(DependencyEdge {
    from: PathBuf::from("src/main.rs"),
    to: PathBuf::from("src/utils.rs"),
    dep_type: DependencyType::Import,
    symbol: None,
});

// Find files affected by a change to utils.rs
let changed = HashSet::from([PathBuf::from("src/utils.rs")]);
let affected = graph.find_affected_files(&changed);
assert!(affected.contains(&PathBuf::from("src/main.rs")));

Runtime Backend Selection

use thread_flow::incremental::{create_backend, BackendType, BackendConfig};

// Select backend based on deployment environment
let backend = if cfg!(feature = "postgres-backend") {
    create_backend(
        BackendType::Postgres,
        BackendConfig::Postgres {
            database_url: std::env::var("DATABASE_URL")?,
        },
    ).await?
} else if cfg!(feature = "d1-backend") {
    create_backend(
        BackendType::D1,
        BackendConfig::D1 {
            account_id: std::env::var("CF_ACCOUNT_ID")?,
            database_id: std::env::var("CF_DATABASE_ID")?,
            api_token: std::env::var("CF_API_TOKEN")?,
        },
    ).await?
} else {
    // Fallback to in-memory for testing
    create_backend(BackendType::InMemory, BackendConfig::InMemory).await?
};

Persistent Storage with Incremental Updates

use thread_flow::incremental::{
    create_backend, BackendType, BackendConfig,
    StorageBackend, AnalysisDefFingerprint, DependencyGraph,
};
use std::path::Path;

async fn incremental_analysis(backend: &dyn StorageBackend) -> Result<(), Box<dyn std::error::Error>> {
    // Load previous dependency graph
    let mut graph = backend.load_full_graph().await?;

    // Check if file changed
    let file_path = Path::new("src/main.rs");
    let new_fp = AnalysisDefFingerprint::new(b"new content");
     
    if let Some(old_fp) = backend.load_fingerprint(file_path).await? {
        if !old_fp.content_matches(b"new content") {
            // File changed - invalidate and re-analyze
            let affected = graph.find_affected_files(&[file_path.to_path_buf()].into());
            for affected_file in affected {
                // Re-analyze affected files...
            }
        }
    }

    // Save updated state
    backend.save_fingerprint(file_path, &new_fp).await?;
    backend.save_full_graph(&graph).await?;
    Ok(())
}

Migration Guide

From Direct Storage Usage to Backend Factory

Before (direct backend instantiation):

#[cfg(feature = "postgres-backend")]
use thread_flow::incremental::backends::postgres::PostgresIncrementalBackend;

let backend = PostgresIncrementalBackend::new(database_url).await?;

After (factory pattern):

use thread_flow::incremental::{create_backend, BackendType, BackendConfig};

let backend = create_backend(
    BackendType::Postgres,
    BackendConfig::Postgres { database_url },
).await?;

Feature Flag Configuration

CLI deployment (Postgres):

[dependencies]
thread-flow = { version = "*", features = ["postgres-backend", "parallel"] }

Edge deployment (D1):

[dependencies]
thread-flow = { version = "*", features = ["d1-backend", "worker"] }

Testing (InMemory):

[dev-dependencies]
thread-flow = { version = "*" }  # InMemory always available

Re-exports

pub use analyzer::AnalysisResult;

pub use analyzer::AnalyzerError;

pub use analyzer::IncrementalAnalyzer;

pub use graph::DependencyGraph;

pub use invalidation::InvalidationDetector;

pub use invalidation::InvalidationError;

pub use invalidation::InvalidationResult;

pub use types::AnalysisDefFingerprint;

pub use types::DependencyEdge;

pub use types::DependencyStrength;

pub use types::DependencyType;

pub use types::SymbolDependency;

pub use types::SymbolKind;

pub use backends::BackendConfig;

pub use backends::BackendType;

pub use backends::IncrementalError;

pub use backends::create_backend;

pub use storage::InMemoryStorage;

pub use storage::StorageBackend;

pub use storage::StorageError;

pub use backends::PostgresIncrementalBackend;

Modules

analyzer

Core incremental analysis coordinator (Phase 4.1).

backends

Concrete storage backend implementations for the incremental update system.

concurrency

Concurrency abstraction layer for incremental analysis.

dependency_builder

Dependency graph builder that coordinates language-specific extractors.

extractors

Dependency extractors for various programming languages.

graph

Dependency graph construction and traversal algorithms.

invalidation

Invalidation detection and topological sorting for incremental updates.

storage

Storage trait definitions for persisting dependency graphs and fingerprints.

types

Core data structures for the incremental update system.