scryer-engine

Core AST parsing, syntax analysis, and symbol resolution engine for Scryer.

Responsibilities

• Workspace Traversal & Gitignore Filtering (WorkspaceScanner): High-throughput file tree walking via ignore::WalkBuilder respecting .gitignore, .ignore, global git filters, and hidden files.
• Incremental Change Detection (ChangeDetector): 256-bit BLAKE3 content hashing compared against cached hashes in Turso (SourceFile.content_hash) to avoid redundant parsing.
• Multi-Language Tree-sitter AST Extraction:
  • Rust (RustAstParser): Extracts declarations (functions, structs, enums, unions, traits, impls, modules, type aliases, constants, statics), condensed signatures, visibility modifiers (public, crate, private), doc comments, and nested lexical scopes.
  • Python (PythonAstParser): Extracts functions, async functions, classes, methods, top-level constants/variables, signatures with type annotations, docstrings (triple/single-quoted), visibility by convention (public, private), and indentation-based scopes.
  • TypeScript / TSX (TypeScriptAstParser): Extracts interfaces, type aliases, classes, enums, functions, arrow function assignments, methods, JSDoc comments, access modifiers (public, private, protected), and lexical scopes.
• Decoupled Batch Ingestion (BatchIngestionActor): Streams parsed entities from Rayon CPU workers into Tokio writer tasks over bounded MPSC channels, executing transactional Turso writes with batch grouping and status-aware record cleanup to prevent dangling references or orphaned symbols.
• Debounced Workspace Watcher (WorkspaceWatcher): One notify-debouncer-full watcher (200ms debounce, one inotify instance) shared by any number of projects (add_project / remove_project). Only non-ignored directories are watched, each non-recursively (WorkspaceScanner::scan_dirs), and watches are reference-counted across nested projects. A file event re-indexes or removes that file in every project containing it. A directory create, delete or rename, or an ignore-file change, re-walks the project's directories and runs an incremental index_project. Events pass through IgnoreFilter, which applies the scanner's rules (hidden files, nested .ignore/.gitignore, .git/info/exclude, global excludes) per path and reloads a directory's rules when its ignore file changes.
• Dependency Discovery (DependencyProvider, CargoDependencyProvider): Pluggable ecosystem abstraction for package discovery. CargoDependencyProvider executes cargo_metadata to discover workspace direct and transitive dependencies, classifies sources (crates.io, git, path), extracts package checksums from Cargo.lock, maps active features, and locates unpacked crate root directories in $CARGO_HOME/registry/src/ or $CARGO_HOME/git/checkouts/.
• High-Level Coordination (EngineService): Unified facade coordinating scanning, hashing, parallel parsing, and database updates.

Architecture

Filesystem (Scanner / Watcher)
      │
      ▼
BLAKE3 ChangeDetector (vs Turso SourceFile.content_hash)
      │ (Added / Modified)
      ▼
Rayon Worker Pool ──► RustAstParser (Tree-sitter)
      │
      ▼ (ParsedFilePayload)
Bounded MPSC Channel (128 capacity)
      │
      ▼
Tokio Ingestion Actor ──► Turso Database Transaction
                           - Cascading delete obsolete records
                           - Insert SourceFile, Scope, Symbol

Usage Example

use std::path::Path;
use scryer_db::ScryerDb;
use scryer_engine::EngineService;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let db = ScryerDb::connect("turso::memory:").await?;
    let engine = EngineService::new(db);

    let project_id = 1;
    let root = Path::new("/path/to/project");

    // Full or incremental indexing pass
    let report = engine.index_project(project_id, root).await?;
    println!(
        "Indexed {} files (added: {}, modified: {}, unchanged: {}, references: {}, edges: {})",
        report.scanned_files,
        report.added_files,
        report.modified_files,
        report.unchanged_files,
        report.total_references,
        report.total_edges,
    );

    // Resolve definition at location
    let def = engine
        .resolve_definition(project_id, root, Path::new("src/main.rs"), 10, 15)
        .await?;
    if let Some(target) = def {
        println!("Resolved {} to {:?}", target.symbol_name, target.file_path);
    }

    // Start background file watcher (200ms debounce); add more with watcher.add_project
    let watcher_handle = engine.watch_project(project_id, root)?;

    // ... application runs ...

    watcher_handle.stop().await;
    Ok(())
}

Stack Graphs & Scope Resolution

scryer-engine provides two-tiered cross-file symbol resolution and call graph linking:

1. Tier 1: Tree-sitter Stack Graphs (StackGraphEngine):

   • Construct declarative name-binding graphs via vendored Tree-sitter Graph (.tsg) rules.

   • Incrementally precompute minimal partial paths per file (ForwardPartialPathStitcher).

   • Resolves references by stitching partial paths across files into complete definition paths.

   • Enforces a deterministic traversal bound (max_steps = 1000) with cancellation checks to mitigate circular imports and path explosions.

   • Re-adding a changed file (or removing one) marks the graph stale; it is rebuilt from stored sources before its next use, since stack graphs can't drop a file's nodes.

2. Tier 2: Syntactic Heuristics Fallback (ScmFallbackResolver):

   • For constructs outside TSG rule coverage (complex macros, closures, dynamic dispatch), evaluates enclosing lexical scopes and workspace qualified symbol matching.
   • Path-qualified Rust references (crate::a::f, other_crate::m::f, Type::new) skip both tiers: normalize_rust_path rewrites crate/self/super, and ingest matches the path against qualified names (find_symbol_for_target).
   • Anything still unresolved is linked by bare name against the persisted index at ingest time, so references into files outside the current indexing pass still link.

Rust qualified names follow the module path (crates/my-pkg/src/a.rs → my_pkg::a; a root-level src/ uses crate). ingest_batch inserts every file's symbols before linking references, and re-indexing a file re-points references from other files at its new symbol IDs.

3. Multi-Project Engine Pool (ProjectEnginePool):
   • Thread-safe LRU cache maintaining active StackGraphEngine instances per tenant project_id.
   • Protects memory footprint while isolating project graph state.