polymathy 0.2.0

Turn search results into answers - a web service that fetches, chunks, and returns semantic content from search queries
Documentation
# Module Structure

Polymathy follows a clean, modular architecture with clearly defined responsibilities.

## Module Overview

```
polymathy/
├── lib.rs      # Library root and public exports
├── api.rs      # HTTP server and endpoints
├── search.rs   # Search data structures
├── index.rs    # Vector index management
└── bin/
    └── polymath.rs  # Binary entry point
```

## api.rs

The main API module handling HTTP requests and responses.

### Key Functions

#### `run()`

```rust
pub async fn run() -> std::io::Result<()>
```

Initializes and starts the Actix web server.

**Responsibilities:**
- Load environment variables
- Configure logging
- Set up HTTP routes
- Start the server

#### `search_and_index()`

```rust
async fn search_and_index(
    query: web::Query<SearchQuery>,
) -> actix_web::Result<HttpResponse>
```

Main search endpoint handler.

**Flow:**
1. Extract query parameter
2. Call SearxNG
3. Process results
4. Return chunk map

#### `process_search_results()`

```rust
async fn process_search_results(
    search_results: Value,
    index: Arc<Mutex<Index>>
) -> Result<HashMap<u64, (String, String)>>
```

Processes search results by fetching and chunking content.

**Key Features:**
- Concurrent URL processing
- Error handling per URL
- Thread-safe chunk collection

## search.rs

Data structures for search operations.

### Structures

#### `SearchQuery`

```rust
#[derive(Deserialize, JsonSchema)]
pub struct SearchQuery {
    pub q: String,
}
```

Represents an incoming search query.

#### `ProcessedContent`

```rust
#[derive(Deserialize)]
pub struct ProcessedContent {
    pub url: String,
    pub config: Config,
    pub chunks: HashMap<String, String>,
    pub embeddings: HashMap<String, Vec<f32>>,
    pub error: Option<String>,
}
```

Response from the content processor service.

#### `Config`

```rust
#[derive(Deserialize)]
pub struct Config {
    pub chunking_type: String,
    pub chunking_size: u32,
    pub embedding_model: String,
}
```

Configuration for content processing.

## index.rs

Vector index management using USearch.

### Functions

#### `create_index()`

```rust
pub fn create_index() -> Index
```

Creates a new USearch vector index.

**Configuration:**
```rust
IndexOptions {
    dimensions: 384,           // AllMiniLML6V2 embedding size
    metric: MetricKind::IP,    // Inner Product
    quantization: ScalarKind::F32,
    connectivity: 16,
    expansion_add: 128,
    expansion_search: 64,
}
```

## lib.rs

Library root that exports public API.

### Public Exports

```rust
pub mod search;
pub mod index;
pub mod api;

pub use api::run;
```

## bin/polymath.rs

Minimal binary entry point.

```rust
use polymathy::run;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    run().await
}
```

## Module Dependencies

```mermaid
graph TD
    A[lib.rs] --> B[api.rs]
    A --> C[search.rs]
    A --> D[index.rs]
    B --> C
    B --> D
    E[bin/polymath.rs] --> A
```

## External Dependencies

| Module | Key Dependencies |
|--------|------------------|
| api.rs | actix-web, reqwest, futures, apistos |
| search.rs | serde, schemars |
| index.rs | usearch |