polymathy 0.2.0

Turn search results into answers - a web service that fetches, chunks, and returns semantic content from search queries
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105

# Architecture Overview

Polymathy is designed as a high-performance, async web service built with Rust.

## System Architecture

```mermaid
graph LR
    A[Client] --> B[Polymathy API]
    B --> C[SearxNG]
    B --> D[Content Processor]
    D --> E[Embedding Model]
    B --> F[USearch Index]
```

## Components

### API Layer

The HTTP server handles incoming requests, processes them, and returns responses. Built with Actix-web for high performance.

**Responsibilities:**
- Accept search queries
- Coordinate with external services
- Return processed results
- Serve API documentation

### Search Integration

Polymathy uses SearxNG as a meta-search engine to retrieve initial search results.

**Flow:**
1. Receive query from client
2. Forward to SearxNG with JSON format
3. Extract URLs from results (max 10)
4. Pass URLs to content processor

### Content Processor

An external service that handles the heavy lifting of content processing.

**Functions:**
- Fetch web page content
- Clean and extract text
- Split into semantic chunks
- Generate vector embeddings

### Vector Index

USearch provides fast vector similarity search capabilities.

**Configuration:**
- 384 dimensions (AllMiniLML6V2)
- Inner Product metric
- F32 quantization

## Data Flow

```mermaid
sequenceDiagram
    participant C as Client
    participant P as Polymathy
    participant S as SearxNG
    participant PR as Processor

    C->>P: GET /v1/search?q=query
    P->>S: Search request
    S-->>P: Search results (URLs)
    loop For each URL (max 10)
        P->>PR: Process URL
        PR-->>P: Chunks + Embeddings
    end
    P-->>C: Processed chunks map
```

## Technology Choices

| Component | Technology | Reason |
|-----------|------------|--------|
| Language | Rust | Performance, safety, async support |
| Web Framework | Actix-web | High performance, mature ecosystem |
| Async Runtime | Tokio | Industry standard for Rust async |
| Vector DB | USearch | Fast, lightweight, Rust bindings |
| HTTP Client | Reqwest | Async-friendly, well-maintained |

## Concurrency Model

Polymathy processes multiple URLs concurrently using Tokio's async runtime:

```rust
// Simplified concurrent processing
let futures: Vec<_> = urls
    .iter()
    .map(|url| process_url(url))
    .collect();

let results = join_all(futures).await;
```

This allows efficient handling of I/O-bound operations like fetching web content.

## Next Steps

- [Modules](modules.md) - Detailed module documentation
- [Data Flow](data-flow.md) - In-depth data flow analysis