# scryer-engine
Core AST parsing, syntax analysis, and symbol resolution engine for Scryer.
## Responsibilities
- **Workspace Traversal & Gitignore Filtering (`WorkspaceScanner`)**: High-throughput file tree walking via `ignore::WalkBuilder` respecting `.gitignore`, `.ignore`, global git filters, and hidden files. Source files over `MAX_SOURCE_FILE_BYTES` (2 MiB; generated and minified bundles) are skipped, and a watched file that grows past it is dropped from the index.
- **Incremental Change Detection (`ChangeDetector`)**: 256-bit BLAKE3 content hashing compared against cached hashes in Turso (`SourceFile.content_hash`) to avoid redundant parsing.
- **Multi-Language Tree-sitter AST Extraction**:
- **Rust (`RustAstParser`)**: Extracts declarations (functions, structs, enums, unions, traits, impls, modules, type aliases, constants, statics), condensed signatures, visibility modifiers (`public`, `crate`, `private`), doc comments, and nested lexical scopes.
- **Python (`PythonAstParser`)**: Extracts functions, async functions, classes, methods, top-level constants/variables, signatures with type annotations, docstrings (triple/single-quoted), visibility by convention (`public`, `private`), and indentation-based scopes.
- **TypeScript / TSX (`TypeScriptAstParser`)**: Extracts interfaces, type aliases, classes, enums, functions, arrow function assignments, methods, JSDoc comments, access modifiers (`public`, `private`, `protected`), and lexical scopes.
- **Decoupled Batch Ingestion (`BatchIngestionActor`)**: Streams parsed entities from Rayon CPU workers into Tokio writer tasks over bounded MPSC channels, executing transactional Turso writes with batch grouping and status-aware record cleanup to prevent dangling references or orphaned symbols.
- **Debounced Workspace Watcher (`WorkspaceWatcher`)**: One `notify-debouncer-full` watcher (200ms debounce, one inotify instance) shared by any number of projects (`add_project` / `remove_project`). Only non-ignored directories are watched, each non-recursively (`WorkspaceScanner::scan_dirs`), and watches are reference-counted across nested projects. A file event re-indexes or removes that file in every project containing it. A directory create, delete or rename, or an ignore-file change, re-walks the project's directories and runs an incremental `index_project`. Events pass through `IgnoreFilter`, which applies the scanner's rules (hidden files, nested `.ignore`/`.gitignore`, `.git/info/exclude`, global excludes) per path and reloads a directory's rules when its ignore file changes.
- **Dependency Discovery (`DependencyProvider`, `CargoDependencyProvider`)**: Pluggable ecosystem abstraction for package discovery. `CargoDependencyProvider` executes `cargo_metadata` to discover workspace direct and transitive dependencies, classifies sources (`crates.io`, `git`, `path`), extracts package checksums from `Cargo.lock`, maps active features, and locates unpacked crate root directories in `$CARGO_HOME/registry/src/` or `$CARGO_HOME/git/checkouts/`.
- **High-Level Coordination (`EngineService`)**: Unified facade coordinating scanning, hashing, parallel parsing, and database updates.
- **Lexical Search (`search`)**: Deterministic BM25F symbol search behind `EngineService::search_symbols` (see below). The `Bm25Index` and tokenizer are generic and reused by `scryer-mcp` to rank ADRs.
## Lexical Search
ADR 0010 chose in-memory lexical ranking over neural embeddings: turso_core 0.7.2 has no FTS or vector index, and embeddings would need a model or network API.
- **Tokenizer (`search::tokenize`)**: splits on non-alphanumerics, camelCase / acronym (`HTTPServerError` → `http server error`) and letter↔digit boundaries; lowercases; strips a trailing `s` from tokens longer than 3 chars; drops language keywords and common English stopwords. Identifiers also emit their whole lowercased form, never stopword-filtered, so exact names rank highest. A query made only of stopwords (`"impl"`, `"to the"`) falls back to its raw lowercased split.
- **Index (`search::bm25`)**: `Bm25Index<D>` with weighted fields, BM25F scoring (k1 = 1.2, b = 0.75), deterministic ties (insertion order), and `matched_terms` per hit.
- **Symbol documents**: name ×3, module path ×2, signature ×1, docstring ×1, file path ×1. `reexport` symbols are skipped.
- **Cache (`search::cache::SearchIndexCache`)**: LRU of 16 indexes keyed by `(project_id, dependency_package_id)`. Every index write bumps the project's generation (`touch_project`, `purge_project`, and dependency indexing for project 0 and for new `project_dependency` links); a stale index is rebuilt wholesale on its next query, since symbol IDs change on every re-ingest. Rows load under the DB lock; tokenization runs on the Rayon pool under `spawn_blocking`; a per-key lock prevents duplicate builds.
- **Scopes (`search::SearchScope`)**: `Project`, or `Dependencies { crate_name }`, which is limited to the project's linked packages (a named crate falls back to any cached version). One package uses its own index; several share the project-0 index with a package filter.
```rust
use scryer_engine::search::{SearchQuery, SearchScope};
let results = engine
.search_symbols(project_id, SearchQuery {
text: Some("retry backoff".into()),
kinds: vec!["fn".into()],
scope: SearchScope::Project,
limit: 20,
..SearchQuery::default()
})
.await?;
for hit in results.hits {
println!("{:.2} {} {}:{}", hit.score, hit.qualified_name, hit.file_path, hit.start_line);
}
```
## Architecture
```
Filesystem (Scanner / Watcher)
│
▼
BLAKE3 ChangeDetector (vs Turso SourceFile.content_hash)
│ (Added / Modified)
▼
Rayon Worker Pool ──► RustAstParser (Tree-sitter)
│
▼ (ParsedFilePayload)
Bounded MPSC Channel (128 capacity)
│
▼
Tokio Ingestion Actor ──► Turso Database Transaction
- Cascading delete obsolete records
- Insert SourceFile, Scope, Symbol
```
## Usage Example
```rust
use std::path::Path;
use scryer_db::ScryerDb;
use scryer_engine::EngineService;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let db = ScryerDb::connect("turso::memory:").await?;
let engine = EngineService::new(db);
let project_id = 1;
let root = Path::new("/path/to/project");
// Full or incremental indexing pass
let report = engine.index_project(project_id, root).await?;
println!(
"Indexed {} files (added: {}, modified: {}, unchanged: {}, references: {}, edges: {})",
report.scanned_files,
report.added_files,
report.modified_files,
report.unchanged_files,
report.total_references,
report.total_edges,
);
// Resolve definition at location
let def = engine
.resolve_definition(project_id, root, Path::new("src/main.rs"), 10, 15)
.await?;
if let Some(target) = def {
println!("Resolved {} to {:?}", target.symbol_name, target.file_path);
}
// Start background file watcher (200ms debounce); add more with watcher.add_project
let watcher_handle = engine.watch_project(project_id, root)?;
// ... application runs ...
watcher_handle.stop().await;
Ok(())
}
```
## Stack Graphs & Scope Resolution
`scryer-engine` provides two-tiered cross-file symbol resolution and call graph linking:
1. **Tier 1: Tree-sitter Stack Graphs (`StackGraphEngine`)**:
- Construct declarative name-binding graphs via vendored Tree-sitter Graph (`.tsg`) rules.
- Incrementally precompute minimal partial paths per file (`ForwardPartialPathStitcher`).
- Resolves references by stitching partial paths across files into complete definition paths.
- Enforces a deterministic traversal bound (`max_steps = 1000`) with cancellation checks to mitigate circular imports and path explosions.
- Re-adding a changed file (or removing one) marks the graph stale; it is rebuilt from stored sources before its next use, since stack graphs can't drop a file's nodes.
2. **Tier 2: Syntactic Heuristics Fallback (`ScmFallbackResolver`)**:
- For constructs outside TSG rule coverage (complex macros, closures, dynamic dispatch), evaluates enclosing lexical scopes and workspace qualified symbol matching.
- Path-qualified Rust references (`crate::a::f`, `other_crate::m::f`, `Type::new`) skip both tiers: `normalize_rust_path` rewrites `crate`/`self`/`super`, and ingest matches the path against qualified names (`find_symbol_for_target`).
- Anything still unresolved is linked by bare name against the persisted index at ingest time, so references into files outside the current indexing pass still link.
Rust qualified names follow the module path (`crates/my-pkg/src/a.rs` → `my_pkg::a`; a root-level `src/` uses `crate`). `ingest_batch` inserts every file's symbols before linking references, and re-indexing a file re-points references from other files at its new symbol IDs.
3. **Multi-Project Engine Pool (`ProjectEnginePool`)**:
- Thread-safe LRU cache maintaining active `StackGraphEngine` instances per tenant `project_id`.
- Protects memory footprint while isolating project graph state.