# Monocle Architecture
This document describes the architecture of the `monocle` project: a BGP information toolkit that can be used as both a Rust library, a command-line application, and an HTTP/SSE server.
## Goals and Design Principles
1. **Library-first**: the core capability lives in the library; the CLI is a thin wrapper.
2. **Clear separation of concerns**:
- persistence and caching in `database/`
- domain operations in `lens/`
- presentation and UX concerns in the CLI (`bin/`)
3. **Extensible**: new functionality should be added as new lenses (and optionally wired into the CLI).
4. **Composability**: lenses should be usable programmatically and in batch/automation contexts.
## Layering Rules
The codebase follows strict layering rules to maintain separation of concerns:
### Repository Layer (`database/`)
Repositories are **data access only**:
- CRUD operations (Create, Read, Update, Delete)
- Query methods that return raw data
- No business logic or policy decisions
- No output formatting
### Lens Layer (`lens/`)
Lenses contain **business logic and policy**:
- Interpretation of data (e.g., RPKI validation logic)
- Coordination between multiple repositories
- Output formatting
- Cache refresh decisions
### CLI Layer (`bin/`)
CLI commands are **thin wrappers**:
- Argument parsing
- Output format selection
- Progress display
- Error presentation
## High-level Architecture
Monocle is organized into three primary layers:
- **CLI layer** (`src/bin/`):
- parses flags/arguments
- loads configuration
- selects output format
- calls into lenses
- **Lens layer** (`src/lens/`):
- provides "use-case" APIs (e.g., search, parse, RPKI lookups)
- controls output shaping via a unified `OutputFormat`
- uses `database/` and external libraries as needed
- **Database layer** (`src/database/`):
- manages storage (SQLite), schema initialization, and caching primitives
- contains repositories for specific datasets
## Directory Structure
```
src/
├── lib.rs # Library entry point / exports
├── config.rs # Configuration and shared status helpers
│
├── database/ # Persistence + caching
│ ├── mod.rs
│ ├── README.md
│ │
│ ├── core/ # Connection + schema
│ │ ├── mod.rs
│ │ ├── connection.rs
│ │ └── schema.rs
│ │
│ ├── session/ # Ephemeral / per-run databases
│ │ ├── mod.rs
│ │ └── msg_store.rs
│ │
│ └── monocle/ # Main persistent monocle database
│ ├── mod.rs
│ ├── asinfo.rs # Unified AS information (from bgpkit-commons)
│ ├── as2rel.rs # AS relationships
│ ├── rpki.rs # ROAs/ASPAs cache (SQLite with blob prefixes)
│ └── pfx2as.rs # Prefix-to-ASN mappings (SQLite with blob prefixes)
│
├── lens/ # Business logic ("use-cases")
│ ├── mod.rs
│ ├── README.md
│ ├── utils.rs # OutputFormat, formatting helpers
│ ├── country.rs # Country code/name lookup
│ │
│ ├── as2rel/ # AS relationship lens
│ │ ├── mod.rs
│ │ ├── args.rs
│ │ └── types.rs
│ │
│ ├── inspect/ # Unified AS/prefix inspection
│ │ ├── mod.rs # InspectLens implementation
│ │ └── types.rs # Result types, section selection
│ │
│ ├── ip/ # IP information lookup
│ │ └── mod.rs
│ │
│ ├── parse/ # MRT file parsing
│ │ └── mod.rs
│ │
│ ├── pfx2as/ # Prefix-to-ASN mapping types
│ │ └── mod.rs # Types only; repository handles lookups
│ │
│ ├── rpki/ # RPKI validation and data
│ │ ├── mod.rs # RpkiLens with validation logic
│ │ └── commons.rs # bgpkit-commons integration
│ │
│ ├── search/ # BGP message search
│ │ ├── mod.rs
│ │ └── query_builder.rs
│ │
│ └── time/ # Time parsing and formatting
│ └── mod.rs
│
├── server/ # HTTP/SSE server (requires `server` feature)
│ ├── mod.rs # Server state, startup, /health, auth wiring
│ ├── http.rs # REST router, API error types, system/info
│ ├── search.rs # SSE search streaming handler
│ ├── auth.rs # Token auth middleware
│ └── rest/ # REST endpoint handlers
│ ├── mod.rs
│ ├── time.rs # time/parse
│ ├── country.rs # country/lookup
│ ├── ip.rs # ip/lookup, ip/public
│ ├── rpki.rs # rpki/roa/lookup, rpki/aspa/lookup, rpki/roa/validate
│ ├── pfx2as.rs # pfx2as/lookup
│ ├── as2rel.rs # as2rel/search, as2rel/relationship, as2rel/refresh
│ ├── inspect.rs # inspect/query
│ └── database.rs # database/status, database/refresh, inspect/refresh
│
└── bin/
├── monocle.rs # CLI entry point
└── commands/ # Command handlers (thin wrappers around lenses)
├── as2rel.rs
├── config.rs # Config display + update, backup, sources
├── country.rs
├── inspect.rs # Unified inspect command (replaces whois, pfx2as)
├── ip.rs
├── parse.rs
├── rpki.rs
├── search.rs
└── time.rs
```
## Key Modules
### `inspect` - Unified AS/Prefix Information
The `inspect` command and lens consolidate multiple data sources into a single query interface:
- **ASInfo**: Core AS data from bgpkit-commons (replaces as2org)
- **Connectivity**: AS2Rel-based upstream/peer/downstream relationships
- **RPKI**: ROAs and ASPA records
- **Pfx2as**: Prefix-to-ASN mappings
Features:
- Auto-detects query type (ASN, prefix, IP address, or name)
- Section selection (`--show basic/prefixes/connectivity/rpki/all`)
- Display limits with `--full`, `--full-roas`, `--full-prefixes`, `--full-connectivity`
- Auto-refresh of stale data
- Multiple output formats
### `rpki` - RPKI Validation
The RPKI lens (`RpkiLens`) provides:
- **Validation logic** (RFC 6811): Valid/Invalid/NotFound states
- **Cache management**: Uses `RpkiRepository` for current data (SQLite with blob prefixes)
- **Historical queries**: Uses bgpkit-commons for date-specific lookups
Layering:
- `RpkiRepository` (database): Raw data access only (CRUD, prefix range queries)
- `RpkiLens` (lens): Validation logic, cache refresh, formatting
### `pfx2as` - Prefix-to-ASN Mapping
The Pfx2as repository (`Pfx2asRepository`) provides:
- **Lookup modes**: Exact, longest prefix match, covering (supernets), covered (subnets)
- **ASN queries**: Get all prefixes for an ASN
- **SQLite storage**: IP prefixes stored as 16-byte start/end address pairs
- **Cache management**: 24-hour TTL with automatic refresh
Note: The file-based cache has been removed; all pfx2as data now uses SQLite.
### `server` - HTTP/SSE API
The HTTP/SSE server (`monocle server`) provides programmatic access to monocle functionality:
- **Protocol**: HTTP REST with JSON request/response bodies
- **Streaming**: SSE (`text/event-stream`) for search with progress, element batches, and terminal events
- **Cancellation**: Client closes HTTP connection; server detects drop via `Arc<AtomicBool>`
- **Backpressure**: Bounded mpsc channel (capacity 32); element batches never dropped
- **Terminal invariant**: Exactly one terminal event (`completed`, `cancelled`, or `error`)
- **Auth**: Optional token-based middleware (`Authorization: Bearer <token>`)
- **DB access**: Per-request `MonocleDatabase` in `spawn_blocking` (connection is `!Send`)
Available endpoint groups:
- `/health`: Health check (always open)
- `/api/v1/system/*`: Server introspection
- `/api/v1/search/*`: SSE streaming BGP search
- `/api/v1/time/*`, `/api/v1/ip/*`, `/api/v1/country/*`: Utility lookups
- `/api/v1/rpki/*`: RPKI ROA/ASPA lookup and validation
- `/api/v1/as2rel/*`: AS relationship queries
- `/api/v1/pfx2as/*`: Prefix-to-ASN mapping
- `/api/v1/inspect/*`: Unified AS/prefix inspection
- `/api/v1/database/*`: Database status and refresh
## Module Architecture
### `config/` (Configuration + Status Reporting)
Responsibilities:
- compute default paths and load config file overrides
- provide shared helpers used by `config` CLI command to display:
- SQLite database info (size, table counts, last update time)
- cache settings and cache directory info
- database management (refresh, backup, sources)
This module is intentionally "infra-ish": it should not implement domain logic.
### `database/` (Persistence and Caching)
The `database` module handles local persistence and caches shared across commands.
#### `database/core/`
Responsibilities:
- create/configure SQLite connections
- define and initialize schema
- expose schema/version checks (used on open)
Notes:
- schema management is owned here; higher-level modules should not issue `CREATE TABLE` etc.
#### `database/monocle/`
Responsibilities:
- main persistent monocle dataset store (SQLite DB under the monocle data directory)
- repositories for datasets:
- ASInfo (unified AS information from bgpkit-commons)
- AS2Rel (AS-level relationships)
- RPKI (ROAs/ASPAs with blob-based prefix storage)
- Pfx2as (prefix-to-ASN mappings with blob-based prefix storage)
- file cache helpers for auxiliary file-based caching
Key idea:
- `MonocleDatabase` is the entry point for accessing the persistent DB/repositories.
#### `database/session/`
Responsibilities:
- short-lived, per-operation SQLite storage (e.g., search results)
- optimized for write-heavy temporary usage and easy export
### `lens/` (Business Logic / Use-cases)
Lenses are the primary public-facing API surface for functionality. A lens:
- takes a `&MonocleDatabase` reference for data access
- defines **argument types** (often serde-serializable; optionally clap-derivable under `cli`)
- defines **result types** (serde-serializable)
- performs the operation (may call into Broker, Parser, SQLite repositories, file caches, etc.)
- emits output using the **unified `OutputFormat`**
#### Output formatting
`lens/utils.rs` contains the global `OutputFormat` used across the CLI to keep formatting consistent and predictable.
#### Progress reporting
Certain lenses support progress callbacks (e.g., parse/search). Progress types are designed to be:
- thread-safe (`Send + Sync`)
- serializable (for GUI or other frontends)
### `bin/` (CLI)
The CLI layer wires together:
- clap argument parsing
- config loading
- output selection (`--format`, `--json`)
- invocation of lens operations
- printing human-readable messages to stderr and data output to stdout (to support piping)
The CLI should not duplicate core logic. It should:
- validate/normalize CLI inputs
- call library APIs
- format/print results
## Typical Data Flows
### CLI flow (conceptual)
1. User runs `monocle <command> ...`
2. CLI parses args and loads `MonocleConfig`
3. CLI determines `OutputFormat`
4. CLI opens `MonocleDatabase` and constructs the lens
5. Lens executes operation:
- uses repository for data access
- applies business logic
- returns typed results (and optionally progress events via callback)
6. CLI prints results via `OutputFormat`
### Library flow (conceptual)
1. Application creates (or opens) the `MonocleDatabase`
2. Application constructs lens with database reference
3. Application calls lens methods
4. Application consumes typed results directly, or uses `OutputFormat` to format for display
### HTTP/SSE flow (conceptual)
1. Client sends HTTP request (POST with JSON body, or GET)
2. Axum router dispatches to handler
3. Handler opens `MonocleDatabase` in `spawn_blocking` (for DB-backed endpoints)
4. Handler calls lens methods, formats result as JSON response
5. For search: handler returns `Sse` stream; worker sends events via bounded channel
6. Client disconnect cancels the search worker via `Arc<AtomicBool>`
## Feature Flags
Monocle supports conditional compilation via Cargo features with a simplified three-tier structure:
### Feature Hierarchy
```
cli (default)
├── server
│ └── lib
└── lib
```
**Quick Guide:**
- **Need the CLI binary?** Use `cli` (includes everything)
- **Need HTTP/SSE server without CLI?** Use `server` (includes lib)
- **Need only library/data access?** Use `lib` (database + all lenses + display)
### Feature Descriptions
| `lib` | Complete library: database + all lenses + display | `rusqlite`, `bgpkit-parser`, `bgpkit-broker`, `tabled`, etc. |
| `server` | HTTP/SSE server (implies `lib`) | `axum`, `tokio`, `serde_json` |
| `cli` | Full CLI binary with progress bars (implies `lib` and `server`) | `clap`, `indicatif` |
### Use Case Scenarios
#### Scenario 1: Library Only
**Features**: `lib`
Use when building applications that need:
- Database operations (SQLite, data loading)
- All lenses (TimeLens, ParseLens, SearchLens, RPKI, Country, InspectLens, etc.)
- Table formatting with tabled
```toml
monocle = { version = "1.1", default-features = false, features = ["lib"] }
```
```rust
use monocle::database::MonocleDatabase;
use monocle::lens::inspect::{InspectLens, InspectQueryOptions};
let db = MonocleDatabase::open_in_dir("~/.local/share/monocle")?;
let lens = InspectLens::new(&db);
let result = lens.query("AS13335", &InspectQueryOptions::default())?;
```
#### Scenario 2: Library with HTTP/SSE Server
**Features**: `server`
Use when building applications that need:
- Everything in `lib`
- HTTP/SSE server for remote API access
```toml
monocle = { version = "1.1", default-features = false, features = ["server"] }
```
```rust
use monocle::config::MonocleConfig;
use monocle::server::start_server;
let config = MonocleConfig::new(&None)?;
// Start HTTP/SSE server on default port
start_server(config).await?;
```
#### Scenario 3: CLI Binary (Default)
**Features**: `cli` (default)
The full CLI binary with all features, HTTP/SSE server, and terminal UI:
```toml
monocle = "1.1"
```
Or explicitly:
```toml
monocle = { version = "1.1", features = ["cli"] }
```
### Valid Feature Combinations
All of these combinations compile successfully:
| (none) | Config types only, no functionality |
| `lib` | Full library functionality |
| `server` | Library + HTTP/SSE server |
| `cli` | Full CLI (includes everything) |
### Feature Dependencies
When you enable a higher-tier feature, lower-tier features are automatically included:
- `server` → automatically enables `lib`
- `cli` → automatically enables `lib` and `server`
## Related Documents
- `README.md` — user-facing CLI and library overview
- `CHANGELOG.md` — version history and breaking changes
- `DEVELOPMENT.md` — contributor guide for adding lenses and fixing bugs
- `src/server/README.md` — HTTP/SSE API specification
- `src/database/README.md` — database module notes
- `src/lens/README.md` — lens module patterns and conventions
- `examples/README.md` — example code organized by feature tier