# Architecture
This document describes the architecture of `oo` — how commands flow through the system and what each module is responsible for.
## Overview
oo wraps shell commands, analyzes their output, and provides context-efficient results to AI agents. The core flow is:
```mermaid
flowchart TD
A[User runs: oo cargo test] --> B[Parse Action]
B --> C{Is reserved subcommand?}
C -->|Yes| D[Execute subcommand<br/>recall/forget/learn/help/init/patterns/rewrite/hook/version]
C -->|No| E[Execute shell command]
E --> F[Capture output<br/>stdout + stderr + exit code]
F --> X{Exit code ≠ 0?}
X -->|Yes| FL["Failure: show filtered slice<br/>(pattern failure strategy or smart_truncate)"]
FL --> FI{"Slice drops content?<br/>(line comparison)"}
FI -->|Yes| FX["Index full output<br/>→ recall line"]
FI -->|No| FN[Return slice]
X -->|No| S{"Output ≤ 4 KiB?"}
S -->|Yes| P["Passthrough verbatim<br/>not indexed"]
S -->|No| G{"Success pattern matches<br/>and yields a summary?"}
G -->|Yes| SU["Success: ✓ summary<br/>not indexed"]
G -->|No| J{Command category}
J -->|Status| ST["Quiet ✓ line<br/>not indexed"]
J -->|Data| LA["Large: index full output<br/>indicator line"]
LA -->|index fails| LT["NOT-indexed header + stderr warning<br/>smart_truncate display"]
J -->|Content / Unknown| BO["Bounded: index full output<br/>head + tail slice"]
```
## Core Flow
### 1. Command Execution
**Module**: [`src/exec.rs`](../src/exec.rs)
Every command (whether a reserved subcommand or shell command) goes through execution:
1. Parse the user's input with `clap`
2. Intercept `_learn_bg` internal commands (for background pattern learning)
3. Dispatch to the appropriate handler:
- Reserved subcommands → built-in handlers
- Everything else → shell command execution
**Key functions**:
- `exec::run(args: &[OsString])` — Execute a shell command and capture stdout, stderr, exit code
- `CommandOutput` — Struct holding captured output
### 2. Output Classification
**Module**: [`src/classify.rs`](../src/classify.rs)
The classification engine decides how to present output to the agent:
**Input**: `CommandOutput` + command string + patterns
**Possible outcomes**:
- **Failure** (exit code ≠ 0): Filtered error output (tail/head/grep/between). When the displayed slice does not match the full merged output line for line (ignoring trailing empty lines), the full output is indexed for recall and a `● label (output truncated: …)` line follows the slice
- **Passthrough** (success, ≤ 4 KiB): Verbatim output
- **Success** (success, > 4 KiB, pattern summary): Compressed summary
- **Status** (success, > 4 KiB, no pattern summary, Status category): quiet `✓ label [saved N]` line (no parentheses), not indexed
- **Large** (success, > 4 KiB, no pattern summary, Data category): Full output indexed, only an indicator line displayed (if indexing fails, a NOT-indexed header, a stderr warning and a `smart_truncate` view are printed instead)
- **Bounded** (success, > 4 KiB, no pattern summary, Content/Unknown category): Full output indexed for recall, only a byte-bounded head+tail slice displayed
**Decision tree**:
1. Exit code zero? No → Failure
2. Output ≤ 4 KiB? Yes → Passthrough
3. Pattern matches and its success strategy yields a summary? Yes → Success. If the matching pattern's success strategy extracts nothing (for example a regex that does not match, or a grep with no matching line), step 4 runs and the category default applies. Only the first matching pattern is considered (first match wins, see [security-model.md](security-model.md)); a later pattern that would match is not tried. A success template that is the empty string yields a Success with an empty summary: the indicator is `✓ label [saved N]` with no parentheses, the same line shape as Status.
4. Detect category → apply category defaults
The **savings indicator** (the `[saved N]` suffix on compressed output) is specified in [docs/cli-reference.md](cli-reference.md#savings-indicator) (canonical).
### 3. Pattern Matching
**Module**: [`src/pattern/`](../src/pattern/)
Patterns define how to compress command output using regex.
**Components**:
- `mod.rs` — Pattern struct, matching logic
- `builtins.rs` — built-in patterns for common tools (list with `oo patterns`)
- `toml.rs` — Load user-defined patterns from the user patterns directory (see `oo patterns` for its path)
**Pattern structure**:
```rust
Pattern {
command_match: Regex, // matches command line (e.g., "cargo test")
success: Option<SuccessPattern>, // how to extract summary on success
failure: Option<FailurePattern>, // how to filter errors on failure
}
```
**Matching priority**: Project patterns (`<git-root>/.oo/patterns/`) → user patterns (`patterns/` in the config directory) → built-in patterns → category defaults
### 4. Category Detection
**Module**: [`src/classify.rs`](../src/classify.rs) (auto-detection)
Commands are auto-categorized to determine default behavior:
| Category | Examples | Default Behavior |
|----------|----------|------------------|
Category definitions and default behaviors: see [Command Categories in the Patterns guide](patterns.md#command-categories) (canonical).
### 5. Storage & Recall
**Module**: [`src/store.rs`](../src/store.rs)
Unpatterned Large (Data) and Bounded (Content/Unknown) outputs are indexed for full-text retrieval, as are the full merged outputs of failed commands whose displayed slice does not match them line for line (see Failure above).
**Storage backends**:
- `SqliteStore` (default) — SQLite database in `dirs::data_dir()/.oo/` (per-OS defaults in [configuration.md](configuration.md#directories); Windows untested). `OO_DATA_DIR` replaces the whole directory when valid (see [configuration.md](configuration.md#directories) for the override rules)
- `VipuneStore` (feature flag) — Optional semantic search via Vipune
**Operations**:
- `index()` — Store output with metadata
- `search()` — Full-text search across indexed outputs
- `delete_project()` — Delete all indexed outputs for the current project
**Project identification**:
- [`src/session.rs`](../src/session.rs) — Project ID (git remote last segment, git root directory name, or current directory name)
### 6. LLM Learning
**Module**: [`src/learn.rs`](../src/learn.rs)
`oo learn` runs a command, observes its output, then generates a pattern via LLM.
**Process**:
1. Execute command normally
2. Capture command, output, exit code
3. Spawn background process
4. Send to Anthropic API (requires `ANTHROPIC_API_KEY`)
5. Parse response as TOML pattern
6. Write to the user patterns directory as `<label>.toml`
**Status tracking**: Background learn results are written to a status file and displayed on the next `oo` invocation.
## Module Responsibilities
| Module | Responsibility | Key Types/Functions |
|--------|---------------|---------------------|
| [`src/main.rs`](../src/main.rs) | CLI entry point, subcommand dispatch | `Cli`, `main()` |
| [`src/exec.rs`](../src/exec.rs) | Shell command execution, output capture | `run(args: &[OsString])`, `CommandOutput` |
| [`src/classify.rs`](../src/classify.rs) | Classification engine, category detection | `classify()`, `detect_category()` |
| [`src/failure_display.rs`](../src/failure_display.rs) | Failure-tier display and the recall notice for failed commands | `render()` |
| [`src/indexing.rs`](../src/indexing.rs) | Index raw output for `oo recall`; shared index-or-warn path | `try_index()`, `index_or_warn()` |
| [`src/display.rs`](../src/display.rs) | Savings suffix for compressed indicator lines | `savings_suffix()` |
| [`src/pattern/mod.rs`](../src/pattern/mod.rs) | Pattern matching, extraction logic | `find_matching()`, `extract_summary()` |
| [`src/pattern/builtins.rs`](../src/pattern/builtins.rs) | Built-in patterns for common tools (list with `oo patterns`) | `BUILTINS`, `builtin_patterns()` |
| [`src/pattern/toml.rs`](../src/pattern/toml.rs) | Load/parse user patterns from TOML | `load_user_patterns()`, `parse_pattern_str()` |
| [`src/store.rs`](../src/store.rs) | Storage backends (SQLite, Vipune) | `Store` trait, `SqliteStore` |
| [`src/session.rs`](../src/session.rs) | Session and project ID detection | `session_id()`, `project_id()` |
| [`src/learn.rs`](../src/learn.rs) | LLM integration, background pattern learning | `run_background(data_path: &Path)`, `LearnConfig` |
| [`src/config.rs`](../src/config.rs) | Config and data directory overrides (`OO_CONFIG_DIR`, `OO_DATA_DIR`) and the platform fallback | `config_dir()`, `data_dir()`, `warn_ignored_overrides()` |
| [`src/retention.rs`](../src/retention.rs) | `[store] retention` parsing and the lazy per-project purge of indexed output | `purge_stale()`, `load_retention()` |
| [`src/error.rs`](../src/error.rs) | Unified error types | `Error` enum |
| [`src/commands.rs`](../src/commands.rs) | CLI command handlers | `cmd_run()`, `cmd_recall()`, etc. |
| [`src/rewrite.rs`](../src/rewrite.rs) | Pure rewriter for `oo rewrite` (agent hooks): quote-aware segment splitting, pattern-based prefixing | `rewrite()` |
| [`src/help.rs`](../src/help.rs) | cheat.sh lookup for `oo help <cmd>` (URL encoding, HTML/error detection) | `lookup()`, `encode_cmd()` |
| [`src/usage.rs`](../src/usage.rs) | Static usage text for reserved subcommands (`oo <sub> --help` and `oo help <sub>`) | `for_subcommand()` |
| [`src/pattern_load.rs`](../src/pattern_load.rs) | Canonical pattern set: project, then user, then builtins | `all_patterns()`, `PATTERNS` |
| [`src/init.rs`](../src/init.rs) | `oo init` hook generation (Claude / generic formats) | Hook file creation |
| [`src/init_pi.rs`](../src/init_pi.rs) | `oo init --agent pi` — embedded pi extension (`OO_TS_EXTENSION`) + idempotent installer (project / global paths) | `run()` |
| [`src/hook.rs`](../src/hook.rs) | `oo hook claude` stdin-JSON PreToolUse processor + `settings.json` merge installer for `oo init --agent claude-code` (tests in [`src/hook_tests.rs`](../src/hook_tests.rs)) | `cmd_hook_claude()`, `merge_oo_hook()` |
| [`src/util.rs`](../src/util.rs) | Utilities (time helpers) | `now_epoch()`, `format_age()` |
## Key Design Decisions
### 1. Bounded passthrough by default
Small successful outputs (≤ 4 KiB) pass through unchanged for every category; failed commands are always filtered (see Failure in the classification outcomes). Bounded Content and Unknown outputs are never dumped verbatim: the full output is indexed for `oo recall` and only a byte-bounded head+tail slice (with a truncation marker) is displayed, so a large `cat`, `git diff`, `jq`, or `sh -c` can no longer blow an agent's context window.
### 2. Patterns are opt-in overrides
For normal command classification, project and user patterns take priority over built-ins: project first, then user, then built-in, with the first matching pattern winning. The foreground display in `oo learn` classifies with built-in patterns only (#151). This lets users customize behavior without modifying oo itself.
### 3. Category-based fallbacks
When no pattern matches, category detection provides sensible defaults. Content commands (git show, diff, cat) and Unknown commands (curl, sh -c, custom scripts) all route large unpatterned output through the Bounded tier — the agent keeps a bounded head+tail view of the actual output while the full content stays recoverable via `oo recall`.
### 4. SQLite by default
SQLite requires no setup, is portable, and provides sufficient full-text search for most use cases. Vipune is an optional upgrade path for semantic search.
### 5. Background learning
Pattern learning runs in the background after the command completes. This doesn't block the user; results appear on the next `oo` invocation.
### 6. Project-scoped storage
Indexed outputs are keyed by a project id. In the default build it is derived from the git remote (last path segment; an empty one falls through), else the git root directory name, else the current directory name. In the `vipune-store` build it comes from vipune's own project detection (`VIPUNE_PROJECT` is honoured first). `oo recall` searches all output with the current project's id, and `oo forget` clears all of it. The id is not a security boundary: repositories that share an id share a partition (see [Security Model](security-model.md#project-scoped-storage)).
## Reserved Subcommands
These are built-in commands (not shell commands):
| Subcommand | Handler | Purpose |
|------------|---------|---------|
| `recall` | `cmd_recall()` | Search indexed outputs via full-text query |
| `forget` | `cmd_forget()` | Clear indexed outputs for the current project (complete for the SQLite store; with `vipune-store` the stop rules and completeness guarantee are in [Security Model](security-model.md#command-output)) |
| `learn` | `cmd_learn()` | Run command and generate pattern via LLM |
| `help` | `cmd_help()` | Show help text or fetch cheat sheet |
| `init` | `cmd_init()` | Set up agent-framework hooks: `.claude/hooks.json` (formats), the pi extension (`--agent pi`), or the Claude Code `settings.json` hook (`--agent claude-code`) |
| `version` | `parse_action()` → `Action::Version` (`oo --version` is handled by clap) | Print version |
| `rewrite` | `cmd_rewrite()` | Print `oo`-prefixed form of a command for agent hooks (never executes) |
| `hook` | `cmd_hook()` | Agent hook processor (e.g. `oo hook claude` reads PreToolUse JSON on stdin) |
| `_learn_bg` | `run_background()` | Internal command for background learning |
Everything else is treated as a shell command to execute.
## Extension Points
### Adding a new built-in pattern
1. Add pattern to `src/pattern/builtins.rs`
2. Write unit tests in the same file
3. Consider adding integration test in `tests/integration.rs`
### Adding a new storage backend
1. Implement `Store` trait in `src/store.rs`
2. Add feature flag to `Cargo.toml`
3. Update storage initialization in main command flow
### Adding a new reserved subcommand
1. Add handler function to `src/commands.rs`
2. Dispatch in `parse_action()` in `src/commands.rs`
3. Add tests in `tests/integration.rs`
## Performance Considerations
- **Output size**: For successful commands, only outputs > 4 KiB are considered for compression/indexing; small ones pass through immediately. Failed commands are classified at any size, and are indexed when their displayed slice does not match the full output line for line.
- **Pattern matching**: Compiled regexes are cached. Pattern matching is O(n) where n is the number of loaded patterns (built-ins listed by `oo patterns`, plus user patterns).
- **Storage**: SQLite uses an FTS5 index over `content`; the `project` column is filtered with a plain predicate (no index on `project`).
- **LLM learning**: Runs in background; doesn't block command execution.
## Security Considerations
See [Security Model](security-model.md) for details on:
- Trust assumptions
- Local data storage
- API key handling
- Command injection prevention
## Library API Stability
The crate `double-o` is released to crates.io by release-plz (`publish = true` in `release-plz.toml`) and ships the `oo` binary. Its library target (`double_o`, `src/lib.rs`) is a public library surface for other Rust code, used for tests and integration. It carries **no stability guarantee** before 1.0: breaking library changes ship in a minor version (0.x → 0.y).
The library surface is every `pub` item reachable from the crate root in `src/lib.rs`. Items marked `#[doc(hidden)]` are part of that surface and follow the same rule. `#[doc(hidden)]` only omits them from the rendered docs; they are not intended for external use (this includes the `argv` and `commands` modules).
Automated checks may not flag every change, so a breaking library change must be marked in its commit (`!` or a `BREAKING CHANGE:` footer).
### Changes to the library surface since v0.9.0
Signatures changed since v0.9.0; downstream code must adapt:
- `exec::run`: `&[String]` → `&[OsString]`
- `learn::run_background`: `data_path: &str` → `data_path: &Path`
- Hidden items in the `commands` module (`double_o::commands::…`): `parse_action`, `run_command_args`, `cmd_run`, and `cmd_learn` take `&[OsString]`; `Action::Run(Vec<OsString>)` and `Action::Learn(Vec<OsString>, Option<String>)`. `parse_action`, `cmd_run`, `cmd_learn` and `Action` are also re-exported from the crate root; `run_command_args` is not.
- New hidden `argv` module.
## Further Reading
- [Testing Guide](testing.md) — How to test each component and testing walkthrough
- [Patterns](patterns.md) — Creating custom patterns
- [Learning Patterns](learn.md) — LLM-based pattern generation