polymathy 0.2.0

Turn search results into answers - a web service that fetches, chunks, and returns semantic content from search queries
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

# API Endpoints

Detailed documentation for all Polymathy API endpoints.

## Search Endpoint

### `GET /v1/search`

Process a search query, retrieve content from results, and return semantic chunks.

#### Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `q` | string | Yes | The search query |

#### Request Example

```bash
curl "http://localhost:8080/v1/search?q=machine+learning+basics"
```

#### Response

Returns a JSON object mapping chunk IDs to `[url, content]` tuples.

```json
{
  "0": ["https://example.com/ml-intro", "Machine learning is a subset of artificial intelligence..."],
  "1": ["https://example.com/ml-intro", "The most common types of machine learning include..."],
  "2": ["https://another-site.com/ml", "Getting started with machine learning requires..."]
}
```

#### Response Fields

| Field | Type | Description |
|-------|------|-------------|
| `key` | string | Sequential chunk ID (0, 1, 2, ...) |
| `value[0]` | string | Source URL where the content was found |
| `value[1]` | string | The content chunk (approximately 100 words) |

#### Processing Flow

```mermaid
sequenceDiagram
    participant C as Client
    participant P as Polymathy
    participant S as SearxNG
    participant PR as Processor

    C->>P: GET /v1/search?q=query
    P->>S: Forward search query
    S-->>P: Return URLs
    loop For each URL (max 10)
        P->>PR: Process URL
        PR-->>P: Return chunks + embeddings
    end
    P-->>C: Return chunk map
```

#### Example: Python Client

```python
import requests

def search(query: str) -> dict:
    response = requests.get(
        "http://localhost:8080/v1/search",
        params={"q": query}
    )
    response.raise_for_status()
    return response.json()

# Usage
results = search("rust async programming")
for chunk_id, (url, content) in results.items():
    print(f"[{chunk_id}] {url}")
    print(f"    {content[:100]}...")
```

#### Example: JavaScript/TypeScript

```typescript
async function search(query: string): Promise<Record<string, [string, string]>> {
  const response = await fetch(
    `http://localhost:8080/v1/search?q=${encodeURIComponent(query)}`
  );

  if (!response.ok) {
    throw new Error(`Search failed: ${response.statusText}`);
  }

  return response.json();
}

// Usage
const results = await search("web development");
Object.entries(results).forEach(([id, [url, content]]) => {
  console.log(`[${id}] ${url}: ${content.slice(0, 100)}...`);
});
```

#### Error Responses

| Status | Cause | Example |
|--------|-------|---------|
| 400 | Missing query parameter | `{"error": "Missing query parameter 'q'"}` |
| 500 | SearxNG unavailable | `{"error": "Failed to fetch search results"}` |
| 500 | Processor unavailable | `{"error": "Content processing failed"}` |

## Rate Limiting

Polymathy itself does not implement rate limiting. Rate limits depend on:

1. Your SearxNG instance configuration
2. Any reverse proxy settings
3. The content processor's capacity

!!! tip "Production Deployments"
    Implement rate limiting at the reverse proxy level for production use.