ratel-ai-core 0.1.5

Tool retrieval and ranking for AI agents — BM25 over tool catalogs. Core of the Ratel context engineering platform.
Documentation

The Rust library at the heart of Ratel. Retrieval, auth, and telemetry are implemented here; every other piece of the project (SDKs, benchmark, future server and integrations) is a wrapper around this crate.

Library shape

  • Crate name: ratel-ai-core
  • Library name: ratel_ai_core
  • In-process; no infra dependencies
  • Member of the root Cargo workspace

Build & test

From the repo root:

cargo build -p ratel-ai-core
cargo test  -p ratel-ai-core
cargo clippy -p ratel-ai-core --all-targets -- -D warnings

Or run against the whole workspace from the repo root with cargo build --workspace / cargo test --workspace.

What gets indexed

Tools are the first content type indexed by the core. Tool search is BM25 over a deterministic flat-text projection of each Tool: its name, description, and a walk of both input_schema and output_schema. Only semantic tokens (property names, descriptions, enum values) are emitted; JSON Schema structure (type, required, $ref, braces, quotes) is skipped. See ADR‑0004 for the algorithm and rationale. The same retrieval primitive carries forward to skills, memories, and message history as those land on the roadmap.

Trace stream

Every layer of Ratel — core, SDK, mcp-server — emits into a single tagged event stream owned by this crate (ADR‑0009). One stream, multiple consumers (inspector, suggestion analyzer, future rerankers and consolidation server), filtered at the consumer.

use std::sync::Arc;
use ratel_ai_core::{JsonlSink, ToolRegistry, TraceEvent};

let sink = Arc::new(JsonlSink::new("session-1", "/tmp/ratel.jsonl")?);
let mut registry = ToolRegistry::with_trace_sink(sink);
registry.register(my_tool);
let _ = registry.search("read a file", 5);
// every `register` and `search` is now appended as one JSON line.

Built-in sinks:

  • NoopSink — default; drops everything.
  • MemorySink — Vec-backed for tests and embedder assertions (snapshot(), drain()).
  • JsonlSink — synchronous O_APPEND per event, mode 0600 on Unix.

Schema: TraceEvent is a tagged enum (search, index_churn, invoke_, gateway_, upstream_, auth_) wrapped in TraceEnvelope { v, ts, session_id, ...event }. The reliability profile is query-log shaped — best-effort, sampleable, lossy on backpressure. See ADR-0009 for the full rationale.

The custom TraceSink trait lets embedders forward events to their own pipeline (HTTP, structured logger, ring buffer). The trait carries a sample_rate() knob (defaulting to 1.0); the rate-limiter implementation is deferred to a later release.