polymathy 0.2.0

Turn search results into answers - a web service that fetches, chunks, and returns semantic content from search queries
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

# Configuration

Polymathy is configured through environment variables. This guide covers all available options.

## Environment Variables

### Required Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `SEARXNG_URL` | URL of your SearxNG search instance | `http://searx.example.com/search` |
| `PROCESSOR_URL` | URL of the content processor service | `http://processor:8081/v1/process` |

### Optional Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `SERVER_HOST` | Host address to bind the server | `127.0.0.1` |
| `SERVER_PORT` | Port to listen on | `8080` |

## Example Configuration

### Development

```env
# .env - Development configuration
SEARXNG_URL=http://localhost:8888/search
PROCESSOR_URL=http://localhost:8081/v1/process
SERVER_HOST=127.0.0.1
SERVER_PORT=8080
```

### Production

```env
# .env - Production configuration
SEARXNG_URL=https://searx.example.com/search
PROCESSOR_URL=http://processor-internal:8081/v1/process
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
```

## Content Processing Configuration

The content processor receives these default settings:

| Setting | Default Value | Description |
|---------|---------------|-------------|
| `chunking_size` | 100 | Number of words per chunk |
| `chunking_type` | `words` | How to split content |
| `embedding_model` | `AllMiniLML6V2` | Model for generating embeddings |

!!! note "Processor Configuration"
    These settings are sent to the content processor service. The processor is responsible for implementing the actual chunking and embedding logic.

## Vector Index Configuration

The USearch vector index is configured with:

| Parameter | Value |
|-----------|-------|
| Dimensions | 384 |
| Metric | Inner Product (IP) |
| Quantization | F32 |
| Connectivity | 16 |
| Expansion Add | 128 |
| Expansion Search | 64 |

## Setting Up SearxNG

Polymathy requires a SearxNG instance for meta-search functionality. Options include:

1. **Self-hosted**: Deploy your own [SearxNG instance](https://docs.searxng.org/)
2. **Public instances**: Use a [public SearxNG instance](https://searx.space/)

!!! warning "Public Instances"
    Public instances may have rate limits or restrictions. For production use, consider self-hosting.

### Minimal SearxNG Configuration

If self-hosting, ensure your `settings.yml` includes:

```yaml
search:
  formats:
    - json
```

## Logging

Polymathy uses `env_logger`. Control log levels with the `RUST_LOG` environment variable:

```bash
# Enable debug logging
RUST_LOG=debug cargo run

# Only show warnings and errors
RUST_LOG=warn cargo run

# Module-specific logging
RUST_LOG=polymathy=debug,actix_web=info cargo run
```