ripex

A language-agnostic parser and fact-extraction library with a CLI. ripex has seven hand-written recursive-descent parser families and eight language modes (JavaScript and TypeScript are distinct modes), built around a common pipeline:

Lexer → Parser → AST → Facts extraction.

ripex is not a compiler, linker, runtime, or type checker. The optional check command delegates validation to the language's existing production toolchain; it does not reimplement compiler semantics.

Supported languages

The in-process parsers are resilient structural parsers optimized for high-fidelity source facts, not replacement compiler front ends. Name resolution, project loading, linking, SDK behavior, and type semantics remain the responsibility of each language's production toolchain. A missing toolchain is reported as unavailable and never treated as success.

All eight language modes parse the complete checked-in corpus without diagnostics. CI also compiles/type-checks the corresponding real fixture projects and requires every language to extract symbols, imports, calls, and variables, with no parser panics or hangs.

Each language pipeline produces four fact types:

• Symbols — functions, classes, structs, enums, traits, interfaces, methods, constants, fields
• Imports — use, import, #include, using, from … import, etc.
• Calls — function calls, method calls, constructor calls, path calls
• Variables — local bindings, parameters, statics, constants with type info

JavaScript and TypeScript also retain hashbang, line, and block comments with precise source spans through ParseResult::comments, including comments at end-of-file. Type-only imports/re-exports, dynamic imports, optional calls, awaited calls, generic call arguments, and destructured bindings are emitted as first-class facts.

The public fact payloads retain the detail needed for lossless downstream indexing: symbol parameters/defaults/docstrings/attributes and type metadata; import specifiers and re-export metadata; awaited, optional, receiver, and generic call metadata; plus variable scope, usage-site, and storage metadata. Consumers can serialize these native facts with the serde feature, then build their own cross-file relationships without discarding parser provenance.

Library

use ripex::{detect_language, parser_for_ext};

// Auto-detect language from extension.
let language = detect_language("src/lib.rs").unwrap();
let parser = parser_for_ext(language.id(), "rs").unwrap();
let result = parser.parse("pub fn add(a: i32, b: i32) -> i32 { a + b }");
let facts = parser.extract(&result);

println!("{} symbols", facts.symbols.len());

External toolchain validation is a separate explicit operation:

use ripex::compiler::{check_with_compiler, CompilerCheckOptions};

let report = check_with_compiler(
    "tests/lang-test/rust/Cargo.toml",
    None,
    &CompilerCheckOptions::default(),
)?;
assert!(report.passed(), "{report:#?}");
# Ok::<(), std::io::Error>(())

The crate exposes:

• LanguageParser trait — parse(), extract(), symbols(), imports(), calls(), variables()
• parser_for(lang_id) / parser_for_ext(lang_id, ext) — language registry
• ExtractionResult — symbols, imports, calls, variables vectors
• Fact types: ParsedSymbol, ParsedImport, ParsedCall, ParsedVariable
• check_with_compiler() / plan_compiler_check() — optional bounded production-toolchain validation
• CompilerCheckReport — per-stage commands, status, exit code, output, and normalized diagnostics

All fact types are serializable when the serde feature is enabled. The cli feature enables serde automatically.

Feature flags

Each language is behind a feature flag, allowing minimal builds:

[dependencies]
ripex = { version = "0.1", default-features = false, features = ["lang-rust", "lang-js"] }

CLI

cargo build

# Parse a file; language auto-detected from extension.
ripex parse path/to/file.rs

# Force a language and emit facts as JSON.
ripex parse src/models/product.js --lang javascript --json

# List supported parsers.
ripex ls

# Optionally validate a file with its production compiler.
ripex check src/lib.rs

# Discover the nearest project manifest and check the whole project.
ripex check src/lib.rs --project --json

# Select a standard and pass include/configuration flags through to the compiler.
ripex check native/main.cpp --standard c++23 --arg=-Iinclude

Flags: --json (versioned machine output), --ast (AST shape summary), --facts (symbol/import/call/variable listing). The process exits non-zero when parse errors are present. JSON includes schema_version, the selected language, structured diagnostics, an AST summary, and all extracted facts.

check exits 0 when every compiler stage passes, 1 when source is rejected, and 2 when a toolchain is unavailable, times out, or cannot be invoked. Each stage has a 120-second default timeout. Project checks can execute compiler plugins, procedural macros, and build scripts; only check trusted projects.

Compiler requirements

ripex discovers these commands on PATH: gcc/clang, g++/clang++, rustc/cargo, go, dotnet/csc, node, tsc, python, and mypy. TypeScript can also be supplied as a local project dependency, discovered via npx --no-install tsc; the checked-in JavaScript fixture pins this dependency. Use --toolchain PATH to override the primary compiler. C/C++ include paths, defines, target triples, SDKs, and other build-specific inputs can be supplied with repeated --arg options. Language-standard conformance is therefore the conformance of the selected production compiler, standard, target, and project configuration rather than a claim that ripex reimplements those compilers.

For a C/C++ directory containing compile_commands.json, check validates each configured translation unit using its recorded compiler, include paths, defines, target flags, and generated-header settings. Output and dependency generation flags are removed and replaced with a no-output semantic check, so the project is not rebuilt or modified. Pass --project when checking one source file and the database is in a parent directory.

Architecture

Each language follows the same module structure:

src/<lang>/
  lexer/        Scanner, token definitions, keyword tables
  parser/       Recursive-descent parser with error recovery
  ast/          Typed AST nodes (statements, expressions, declarations)
  facts.rs      AST → ExtractionResult (the consumer-facing fact extractor)
  semantic/     Scope tracking, symbol tables, binding resolution
  syntax/       Language feature flags, operator precedence
  codegen/      Experimental canonical printers (not source preserving)
  transform/    AST transformation passes (JS pipeline + plugins)
  visit/        Visitor, walker, fold patterns for AST traversal
  diagnostics/  Error types and reporter
  config/       Parser options and configuration
  tests/        Per-language test suites

Safety limits

Built-in guards prevent pathological inputs from causing runaway allocation:

• MAX_INPUT_SIZE: 1 MB
• MAX_TOKENS: 200,000
• MAX_RECURSION: 512 depth

All guards are enforced in the lexer and parser state machines.

Testing

The test suite covers lexer tokenization, parser correctness, feature-isolated builds, every fact category, generator reuse, corpus parse-generate-parse round trips, safety limits, and zero-diagnostic fixture parsing.

cargo test --all-targets --all-features

Verification and Safety

Resource Limits and Boundaries

To prevent denial of service and resource exhaustion, ripex enforces strict limits on all parsed inputs:

• Maximum Input Size: 1 MB (MAX_INPUT_SIZE = 1,048,576 bytes). Enforced at CLI ingestion before allocations occur.
• Maximum Token Count: 200,000 tokens (MAX_TOKENS). Enforced during lexing.
• Maximum Recursion Depth: 512 frames (MAX_RECURSION). Enforced in all recursive-descent parsers to prevent stack overflow.

Breaching any limit produces a structured diagnostic of code limit_exceeded and sets status to LimitExceeded.

Trust Boundary and Compiler Safety

The ripex check command executes external compilers (such as gcc, clang, rustc/cargo, go, dotnet/csc, node, tsc, python, and mypy) on target source code or projects.

• Project Execution Risk: Running compilation check on projects (e.g., Cargo workspace, Go module, C# csproj) can execute arbitrary code via build scripts (build.rs), compiler plugins, or procedural macros.
• Explicit Trust Required: To prevent unauthorized code execution, project-level checks and raw compiler argument passthrough require the --trusted-project flag. If this flag is omitted, the planner immediately returns a PermissionDenied error.
• Sandbox Recommendation: We recommend running ripex check in an OS-level sandbox or containerized environment when validating untrusted third-party code.

License

MIT