scryer-engine

Core AST parsing, syntax analysis, and symbol resolution engine for Scryer.

Responsibilities

• Workspace Traversal & Gitignore Filtering (WorkspaceScanner): High-throughput file tree walking via ignore::WalkBuilder respecting .gitignore, .ignore, global git filters, and hidden files. Source files over MAX_SOURCE_FILE_BYTES (2 MiB; generated and minified bundles) are skipped, and a watched file that grows past it is dropped from the index.
• Incremental Change Detection (ChangeDetector): 256-bit BLAKE3 content hashing compared against cached hashes in Turso (SourceFile.content_hash) to avoid redundant parsing.
• Multi-Language Tree-sitter AST Extraction:
  • Rust (RustAstParser): Extracts declarations (functions, structs, enums, unions, traits, impls, modules, type aliases, constants, statics), condensed signatures, visibility modifiers (public, crate, private), doc comments, and nested lexical scopes.
  • Python (PythonAstParser): Extracts functions, async functions, classes, methods, top-level constants/variables, signatures with type annotations, docstrings (triple/single-quoted), visibility by convention (public, private), and indentation-based scopes.
  • TypeScript / TSX (TypeScriptAstParser): Extracts interfaces, type aliases, classes, enums, functions, arrow function assignments, methods, JSDoc comments, access modifiers (public, private, protected), and lexical scopes.
• Decoupled Batch Ingestion (BatchIngestionActor): Streams parsed entities from Rayon CPU workers into Tokio writer tasks over bounded MPSC channels, executing transactional Turso writes with batch grouping and status-aware record cleanup to prevent dangling references or orphaned symbols.
• Debounced Workspace Watcher (WorkspaceWatcher): One notify-debouncer-full watcher (200ms debounce, one inotify instance) shared by any number of projects (add_project / remove_project). Only non-ignored directories are watched, each non-recursively (WorkspaceScanner::scan_dirs), and watches are reference-counted across nested projects. A file event re-indexes or removes that file in every project containing it. A directory create, delete or rename, or an ignore-file change, re-walks the project's directories and runs an incremental index_project. Events pass through IgnoreFilter, which applies the scanner's rules (hidden files, nested .ignore/.gitignore, .git/info/exclude, global excludes) per path and reloads a directory's rules when its ignore file changes.
• Dependency Discovery (DependencyProvider, CargoDependencyProvider): Pluggable ecosystem abstraction for package discovery. CargoDependencyProvider executes cargo_metadata to discover workspace direct and transitive dependencies, classifies sources (crates.io, git, path), extracts package checksums from Cargo.lock, maps active features, and locates unpacked crate root directories in $CARGO_HOME/registry/src/ or $CARGO_HOME/git/checkouts/.
• High-Level Coordination (EngineService): Unified facade coordinating scanning, hashing, parallel parsing, and database updates.
• Lexical Search (search): Deterministic BM25F symbol search behind EngineService::search_symbols (see below). The Bm25Index and tokenizer are generic and reused by scryer-mcp to rank ADRs.

Lexical Search

ADR 0010 chose in-memory lexical ranking over neural embeddings: turso_core 0.7.2 has no FTS or vector index, and embeddings would need a model or network API.

• Tokenizer (search::tokenize): splits on non-alphanumerics, camelCase / acronym (HTTPServerError → http server error) and letter↔digit boundaries; lowercases; strips a trailing s from tokens longer than 3 chars; drops language keywords and common English stopwords. Identifiers also emit their whole lowercased form, never stopword-filtered, so exact names rank highest. A query made only of stopwords ("impl", "to the") falls back to its raw lowercased split.
• Index (search::bm25): Bm25Index<D> with weighted fields, BM25F scoring (k1 = 1.2, b = 0.75), deterministic ties (insertion order), and matched_terms per hit.
• Symbol documents: name ×3, module path ×2, signature ×1, docstring ×1, file path ×1. reexport symbols are skipped.
• Cache (search::cache::SearchIndexCache): LRU of 16 indexes keyed by (project_id, dependency_package_id). Every index write bumps the project's generation (touch_project, purge_project, and dependency indexing for project 0 and for new project_dependency links); a stale index is rebuilt wholesale on its next query, since symbol IDs change on every re-ingest. Rows load under the DB lock; tokenization runs on the Rayon pool under spawn_blocking; a per-key lock prevents duplicate builds.
• Scopes (search::SearchScope): Project, or Dependencies { crate_name }, which is limited to the project's linked packages (a named crate falls back to any cached version). One package uses its own index; several share the project-0 index with a package filter.

use scryer_engine::search::{SearchQuery, SearchScope};

let results = engine
    .search_symbols(project_id, SearchQuery {
        text: Some("retry backoff".into()),
        kinds: vec!["fn".into()],
        scope: SearchScope::Project,
        limit: 20,
        ..SearchQuery::default()
    })
    .await?;
for hit in results.hits {
    println!("{:.2} {} {}:{}", hit.score, hit.qualified_name, hit.file_path, hit.start_line);
}

Architecture

Filesystem (Scanner / Watcher)
      │
      ▼
BLAKE3 ChangeDetector (vs Turso SourceFile.content_hash)
      │ (Added / Modified)
      ▼
Rayon Worker Pool ──► RustAstParser (Tree-sitter)
      │
      ▼ (ParsedFilePayload)
Bounded MPSC Channel (128 capacity)
      │
      ▼
Tokio Ingestion Actor ──► Turso Database Transaction
                           - Cascading delete obsolete records
                           - Insert SourceFile, Scope, Symbol

Usage Example

use std::path::Path;
use scryer_db::ScryerDb;
use scryer_engine::EngineService;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let db = ScryerDb::connect("turso::memory:").await?;
    let engine = EngineService::new(db);

    let project_id = 1;
    let root = Path::new("/path/to/project");

    // Full or incremental indexing pass
    let report = engine.index_project(project_id, root).await?;
    println!(
        "Indexed {} files (added: {}, modified: {}, unchanged: {}, references: {}, edges: {})",
        report.scanned_files,
        report.added_files,
        report.modified_files,
        report.unchanged_files,
        report.total_references,
        report.total_edges,
    );

    // Resolve definition at location
    let def = engine
        .resolve_definition(project_id, root, Path::new("src/main.rs"), 10, 15)
        .await?;
    if let Some(target) = def {
        println!("Resolved {} to {:?}", target.symbol_name, target.file_path);
    }

    // Start background file watcher (200ms debounce); add more with watcher.add_project
    let watcher_handle = engine.watch_project(project_id, root)?;

    // ... application runs ...

    watcher_handle.stop().await;
    Ok(())
}

Stack Graphs & Scope Resolution

scryer-engine provides two-tiered cross-file symbol resolution and call graph linking:

1. Tier 1: Tree-sitter Stack Graphs (StackGraphEngine):

   • Construct declarative name-binding graphs via vendored Tree-sitter Graph (.tsg) rules.

   • Incrementally precompute minimal partial paths per file (ForwardPartialPathStitcher).

   • Resolves references by stitching partial paths across files into complete definition paths.

   • Enforces a deterministic traversal bound (max_steps = 1000) with cancellation checks to mitigate circular imports and path explosions.

   • Re-adding a changed file (or removing one) marks the graph stale; it is rebuilt from stored sources before its next use, since stack graphs can't drop a file's nodes.

2. Tier 2: Syntactic Heuristics Fallback (ScmFallbackResolver):

   • For constructs outside TSG rule coverage (complex macros, closures, dynamic dispatch), evaluates enclosing lexical scopes and workspace qualified symbol matching.
   • Path-qualified Rust references (crate::a::f, other_crate::m::f, Type::new) skip both tiers: normalize_rust_path rewrites crate/self/super, and ingest matches the path against qualified names (find_symbol_for_target).
   • Anything still unresolved is linked by bare name against the persisted index at ingest time, so references into files outside the current indexing pass still link.

Rust qualified names follow the module path (crates/my-pkg/src/a.rs → my_pkg::a; a root-level src/ uses crate). ingest_batch inserts every file's symbols before linking references, and re-indexing a file re-points references from other files at its new symbol IDs.

3. Multi-Project Engine Pool (ProjectEnginePool):
   • Thread-safe LRU cache maintaining active StackGraphEngine instances per tenant project_id.
   • Protects memory footprint while isolating project graph state.