1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
"""rustai — a zero-cost local research pipeline.
Collect, denoise and compress the open web into something a small local model
can actually read, without an API key and without a heavyweight Python crawler
in the process.
The three things you will use most::
import rustai
# 1. One call: search, read, and compress to a context window.
result = rustai.research("what is BM25", max_sources=5, max_tokens=2048)
print(result.markdown)
# 2. A reusable client, so the connection pool is shared.
client = rustai.Client(providers=["duckduckgo", "wikipedia:en"], max_tokens=4096)
articles = client.read(["https://example.com/post"])
# 3. Offline: turn HTML you already have into clean Markdown.
article = rustai.extract(html, url="https://example.com/post")
articles = rustai.extract_many(list_of_html) # parallel across every core
Listing pages -- front pages, archives, feeds rendered for people -- are
collected as link inventories rather than prose, and work as a source::
front = rustai.extract(html, url) # .kind == "index"
for link in front.links:
print(link.text, link.url)
client = rustai.Client(providers=["index:https://news.ycombinator.com/"])
For a vector store, chunk the articles rather than embedding units directly --
a unit is one paragraph and a third of them are under 30 tokens::
for chunk in article.chunks(target_tokens=512, overlap_tokens=64):
store.add(chunk["text"], metadata=chunk)
chunks = rustai.chunk_many(articles) # parallel across every core
Bring your own embedding model and it ranks alongside BM25 -- this library
does not bundle one, and the arithmetic runs across every core::
units = [u.text for a in articles for u in a.units]
ctx = rustai.slim(
query, articles,
query_vector=model.encode(query).tolist(),
unit_vectors=model.encode(units).tolist(),
semantic_weight=0.5, # 0 is pure BM25, 1 is pure vector
)
Scholarly sources are first-class alongside web search::
client = rustai.Client(
providers=["arxiv", "openalex", "crossref", "europepmc", "duckduckgo"],
contact_email="you@example.com", # OpenAlex/Crossref polite pool
)
Some questions are answered by a conversation rather than a paper::
client = rustai.Client(providers=["hackernews", "stackexchange", "github"])
When a question wants corroboration rather than depth, stop any one page from
taking the whole window::
r = rustai.research("what is BM25 term saturation", max_tokens_per_source=500)
Everything blocking releases the GIL, so these calls parallelise across threads.
"""
=