rustai-core 0.4.0

Zero-cost local research pipeline: async search routing, TLS-fingerprint-aware fetching, SIMD DOM denoising and BM25 context slimming for local LLMs.
Documentation

rustai

Local, zero-cost research for local models. Search the open web, strip it down to the text that matters, and hand a small model a context window it can actually use — with no API key, no per-call billing, and no heavyweight Python crawler in the middle.

import rustai

result = rustai.research("how does BM25 handle document length", max_sources=5, max_tokens=2048)
print(result.markdown)   # cited, deduplicated, ranked Markdown — ready to prompt

CI PyPI crates.io License


Why this exists

Retrieval for a local model is a data-cleaning problem wearing a search costume. The search itself is the easy part; what actually decides whether a 7B model answers well is how much of its 4k window you spent on cookie banners.

Hosted APIs solve this for a fee. Python crawler stacks solve it by pulling in a browser-grade DOM, which is where the memory goes. rustai does it locally in Rust:

Hosted search API Python crawler stack rustai
Cost per query metered free free
API key required — none
Extraction vendor's yours to build built in
Context compression rarely yours to build built in
Scholarly sources rarely yours to build built in
200 docs n/a 0.78s, 55.6 MB 0.11s, 51.9 MB

See Benchmarks for how those numbers were produced.

How it compares

The honest version, because every number below is one this repository measures and you can re-run.

Against extraction libraries — trafilatura, resiliparse, readability-lxml, justext. They do one of the five things this does, and two of them do it well:

rustai trafilatura resiliparse
Extraction F1, 20 real pages 72.6% 75.1% not scored
Serial extraction 64 docs/s 5 docs/s 103 docs/s
Across cores 77 docs/s 4 docs/s 116 docs/s
Search routing 13 provider kinds — —
Ranking to a token budget built in — —
Tokens to answer a question 1,884 18,968 18,968

trafilatura extracts better. Two thirds of its 2.5-point lead is Wikipedia reference lists that it keeps and this library drops — right for recall against a page's own container, wrong for a token budget being spent on an answer — but the remaining third is prose, and that part is a genuine deficit.

resiliparse extracts faster, by 1.6× here. It is C++ and extraction is its only job. The ranking flips with the hardware: on an Apple M1 it was 2.4× ahead, on Colab's x86 pair it has come out both ahead and 5% behind on different runs. Do not trust either number without running it on your machine.

Neither closes the gap that matters. Feeding a question's pages to a model costs 18,968 tokens through the best extractor and 1,884 through this one, with the answer present either way — a tenth, because extraction is not the last stage here. If you want the best article text, use trafilatura. If you want the fewest tokens that still answer the question, that is this library.

Against crawler frameworks — Scrapy, Crawl4AI. They are frameworks: you bring the pipeline and they schedule it. This is a library with one opinion, which is the shape of the output. Not benchmarked here, so no numbers claimed.

Against hosted search and scrape APIs — Tavily, Exa, Firecrawl, Jina Reader. They will out-recall a keyless router because they run their own index. They are also metered, keyed, and see every query you make. This runs on your machine, costs nothing per call, and works offline for the extraction half.

Against HTTP clients — httpx, aiohttp, requests. Fetching fifteen distinct hosts, this library is within noise of the async clients while also reading every robots.txt and honouring the Crawl-delay they ignore. On the same corpus arXiv asks for fifteen seconds between requests; only this client waits.

Try it in Colab

notebooks/colab_extreme.ipynb is the one to open. Nine sections, each measuring something rather than claiming it: a listing page turned into a crawl frontier, concurrency held against itself, thirty pages fetched and extracted with the memory measured, eight provider kinds fused, the token budget checked at six sizes, a benchmark against trafilatura, and the whole pipeline down to a prompt. Installing is one pip line, so the notebook spends its length on the library rather than on a build.

colab_slm_chat.ipynb is the one that answers the question this library exists for: does any of it help a small model? One cell on a T4 — installs, caches a 2–3B Korean-capable model to Google Drive, loads it, and holds a conversation where every answer is grounded in pages fetched seconds earlier. Each question is answered twice by the same model, once from memory and once from retrieved context, because a retrieval library that is never compared against not retrieving is a library nobody has measured.

colab_speed_benchmark.ipynb times this library against five other extractors and three HTTP clients on the same pages, in one cell. It does not flatter us: resiliparse, a C++ extraction library, extracts 2.4× faster on an M1 — and 5% slower on Colab's x86 pair, which is why the notebook tells you to run it on your own machine rather than trust either number. It also separates parsers from extractors and prints output sizes, because comparing the speed of two things that are not doing the same work is how most scraping benchmarks mislead, and it lists rustai twice on the network stage, obeying and ignoring Crawl-delay, since it is the only client there that reads that line.

colab_quickstart.ipynb walks the same ground more slowly, and colab_oneshot.ipynb is a single cell that installs and prints a report. Both predate the PyPI release and still know how to build from source, which is only useful now if you want the browser feature.

Install

pip install rustai

Wheels are published for Linux, macOS and Windows on CPython 3.9+ (a single abi3 wheel per platform). No Rust toolchain needed to install.

For the Rust crate (published as rustai-core, since rustai was taken on crates.io):

cargo add rustai-core

Five minutes

Each step below runs on its own and answers the question the previous one leaves you with. Nothing here needs configuring first.

1 — Ask a question. Search, fetch, clean and compress, in one call.

import rustai

r = rustai.research("how does BM25 handle document length", max_tokens=1024)
print(r.markdown)          # cited Markdown, ready to paste into a prompt

2 — See what it cost, and what it skipped. A retrieval step you cannot account for is one you cannot debug.

print(r.context.tokens)                    # 1017 — under the 1024 you asked for
for src in r.context.sources:
    print(src["tokens"], src["url"])       # who contributed what
for stage, why in r.failures:
    print("skipped:", stage, why)          # dead links say so

The budget is a guarantee, not an estimate: the slimmer renders, measures, drops the weakest unit and repeats until the real total fits.

3 — Clean HTML you already have. No network, no client, no keys — useful for seeing what the denoiser does before you trust it with anything.

article = rustai.extract(html, url="https://example.com/post")

article.markdown              # nav, footer, ads and share widgets gone
article.units                 # the same thing as rankable blocks
article.stats.compression     # how much was dropped

If you pass a URL here by mistake it will tell you so — extract does not fetch. Use step 4 for that.

4 — Bring your own URLs. Read pages, then compress them against a query of your choosing.

client = rustai.Client(contact_email="you@example.com")
articles = client.read([
    "https://en.wikipedia.org/wiki/Okapi_BM25",
    "https://en.wikipedia.org/wiki/Tf%E2%80%93idf",
])

ctx = rustai.slim("document length normalisation", articles, max_tokens=800)
print(ctx.markdown)

read fetches concurrently and drops URLs that failed, so the list you get back may be shorter than the one you passed. Match on Article.url, or pass raise_on_error=True.

5 — Follow a front page. Listing pages come back as link inventories rather than prose, which makes them usable as crawl seeds.

front = client.read(["https://blog.rust-lang.org/"])[0]
front.kind                    # "index"
for link in front.links:
    print(link.text, link.url, link.snippet)

Where to go next: Usage for the full surface, Being a good citizen before you point it at anyone else's server, and the Colab notebook above if you would rather run than read.

What it does, stage by stage

query ──▶ search router ──▶ fetcher ──▶ denoiser ──▶ slimmer ──▶ context
          concurrent        impersonated  DOM heuristics  BM25 + density
          free providers    + polite      + Markdown      + MMR dedupe

1. Search router — free providers, fused

Kind Providers
Web search duckduckgo, searxng:<instance>
Reference wikipedia, wikipedia:ko (any language edition)
Scholarly arxiv, openalex, crossref, europepmc
Community hackernews, stackexchange, stackexchange:<site>, github
Sites you trust rss:<feed>, sitemap:<sitemap.xml>, index:<front page>

All keyless, all free, all queried concurrently; a provider that fails or rate-limits degrades the result set instead of failing the call.

The scholarly providers are not a nicety. A web search for a paper returns blog posts about the paper; arXiv, OpenAlex and Crossref return the paper. OpenAlex stores abstracts as an inverted index for licensing reasons, and rustai reconstructs them, which makes its snippets the most informative of any provider here. arXiv results always point at the abstract page, never the PDF, because a PDF is not something this pipeline can read.

europepmc covers the life sciences — PubMed, PMC and preprints — in one request, where PubMed's own API needs two.

The community providers answer a different kind of question. A paper explains what a method is; a Stack Exchange thread explains why it did not work for somebody, a Hacker News thread explains what practitioners argued about it, and a GitHub result says whether anyone implemented it. stackexchange defaults to Stack Overflow and takes any site key — stackexchange:serverfault, stackexchange:stats. GitHub search is capped at ten requests a minute without a token, which one query per search fits inside.

index: harvests an HTML listing page — a front page, an archive, a forum — for the links it offers. A site with no feed and no sitemap is still collectable.

OpenAlex and Crossref run a faster "polite pool" for callers who identify themselves — pass contact_email to use it.

Rankings are combined with reciprocal rank fusion, because provider scores are not comparable to each other but ranks always are. URLs are canonicalised first — www., tracking parameters, fragments and trailing slashes stripped — so a page found by three providers counts once and ranks higher for it.

2. Fetcher — looks like a browser, behaves like a guest

Bot walls read your TLS ClientHello (JA3/JA4) and HTTP/2 SETTINGS frame, not your User-Agent. rustai replays a real Chrome's via wreq, so ordinary pages return ordinary 200s.

Being able to get in is not a licence to be rude, so robots.txt, Crawl-delay, per-host spacing, a concurrency ceiling, capped bodies and backoff-on-retry are all on by default. Legacy encodings (EUC-KR, Shift_JIS) are decoded properly rather than assumed to be UTF-8.

When a site answers with Retry-After, that is honoured in place of the backoff curve — a server's own number beats any we could invent — up to max_retry_after, past which a delay is really a refusal.

For IP-reputation blocking, which fingerprinting cannot touch, pass proxies and requests rotate round-robin across them. cookie_file persists the jar, so a session — clearance cookies included — survives the process.

HTTP redirects are followed, and so are HTML-level ones: a stub page that redirects through <meta refresh> or location.replace returns a perfectly good 200 containing no content, which is how sites that canonicalise URLs in the browser silently produce empty extractions elsewhere.

For the minority of pages that ship an empty shell and build the DOM in JavaScript, an opt-in headless Chrome fallback fires — but only after a cheap static fetch has demonstrably failed, since rendering costs ~100× a fetch.

3. Denoiser — find the article, drop the furniture

Three signals, in order:

  1. Structural priors — <nav>, <footer>, <aside>, role="banner", and a class/id vocabulary that has been stable for a decade.
  2. Link density — anchor text over total text. Navigation and "related posts" rails approach 1.0; prose approaches 0.
  3. Text-to-HTML ratio — visible characters over serialised markup bytes. Ad slots and widgets are almost pure markup.

Then a Readability-style content score picks the container holding the article, and the survivors are written as clean Markdown — headings, lists, fenced code, pipe tables, absolutised links.

The output is not one blob. It is a list of units, each carrying its own text, its Markdown, its heading breadcrumb and its token cost. That granularity is what makes the next stage possible.

Listing pages take a different path. A front page defeats article extraction for the same reason navigation is boilerplate everywhere else — except here the link density is the content. rustai detects that by measuring how much of the page's prose belongs to a link, and returns an inventory instead: titles, URLs, standfirsts and section headings, ready to fetch. article.kind tells you which you got. On thirteen real pages spanning news front pages, aggregators, encyclopaedia articles, papers, READMEs and specs, the classifier is 12 for 12; it costs about 1% of extraction time and index_mode="never" turns it off.

4. Slimmer — spend the context window deliberately

Given a question and a pile of articles, choose the units that best fill a budget:

  • BM25 relevance at unit granularity, so you get the two paragraphs that answer the question rather than the page that contains them.
  • Information density — content-word ratio, numeric ratio, type-token ratio, minus an explicit boilerplate penalty. This is what separates a dense factual paragraph from "In this article we will explore some of the things you need to know", which matches query terms perfectly and says nothing.
  • MMR selection with an overlap-coefficient redundancy penalty, because three copies of the same syndicated paragraph is the most common way to waste a window.

Output is Markdown with source headers, URLs, heading breadcrumbs and […] elision markers, under the token budget you set.

BM25 is lexical, so it cannot see that "딥러닝" and "deep learning" name one subject, or that a paragraph explains a term without using it. Pass vectors from a model you already run and cosine similarity folds into the same selection — this library bundles no embedding model, because that would trade a keyless install and a fast build for a few hundred megabytes of weights. Swept over nine questions with a multilingual MiniLM, the blend beats both ends: pure BM25 answers 96%, pure cosine 96%, half and half 100%.

5. Or: chunks, if the destination is a vector store

A unit is one block, which is the right grain to rank and the wrong one to embed — on MDN's Promise reference the median unit is 37 tokens and a third are under 30, so embedding units directly gives 125 vectors that each know almost nothing. article.chunks() groups them to a target size, starting a new chunk at each heading so a chunk opens with the thing that says what it is about, and repeating whole units across the boundary so an answer that straddles one stays retrievable from either side. The same page becomes 26 chunks with a median of 409 tokens, each carrying its URL, title and heading path.

Usage

One call

import rustai

r = rustai.research("what changed in HTTP/3", max_sources=5, max_tokens=2048)
print(r.markdown)          # the context
print(r.context.tokens)    # what it actually cost
for src in r.context.sources:
    print(src["url"], src["tokens"])
for stage, message in r.failures:
    print("skipped:", stage, message)   # nothing is swallowed silently

A reusable client

Sharing a client shares its connection pool, its robots.txt cache and its per-host pacing state, so it is meaningfully faster than repeated one-shot calls.

client = rustai.Client(
    providers=["duckduckgo", "wikipedia:en", "arxiv", "openalex"],
    contact_email="you@example.com",   # OpenAlex/Crossref polite pool
    max_tokens=4096,
    concurrency=24,
    impersonate="chrome",       # or "firefox", "safari", "random", "chrome_143", "none"
    respect_robots=True,
    proxies=["socks5://user:pass@host:1080"],   # rotated round-robin
    cookie_file="~/.cache/rustai/jar.json",     # session survives the process
)
...
client.save_cookies()       # write the jar back out

hits = client.search("rust async runtime")           # list[SearchResult]
pages = client.fetch([h.url for h in hits[:5]])      # list[Page]   — raw HTML
articles = client.read([h.url for h in hits[:5]])    # list[Article] — cleaned
result = client.research("rust async runtime")       # the whole pipeline

Offline: clean HTML you already have

No network, no client, nothing to configure:

article = rustai.extract(html, url="https://example.com/post")

article.title            # str | None
article.markdown         # cleaned Markdown
article.text             # plain text
article.tokens           # estimated LLM tokens
article.stats.compression  # e.g. 0.94 — how much was dropped
for unit in article.units:
    print(unit.kind, unit.tokens, unit.heading_path, unit.text[:60])

For a batch, extract_many runs across every core and releases the GIL, so it is roughly 4× the throughput of a loop over extract — and about 30× that of trafilatura:

articles = rustai.extract_many(list_of_html)                 # 1:1 with the input
articles = rustai.extract_many(list_of_html, list_of_urls)   # positional urls

Compress a set you assembled yourself

articles = client.read(my_urls)
ctx = rustai.slim("my question", articles, max_tokens=1024, max_tokens_per_source=400)
print(ctx.markdown)

max_tokens_per_source caps how much any single page can contribute, so one long article cannot crowd out corroborating sources. It is also accepted by research() and Client, so you do not have to drop down to slim to reach it:

r = rustai.research("what is BM25 term saturation", max_tokens_per_source=500)

Leave it off for most questions. Measured over six queries at budgets of 1,000, 2,048 and 4,096 tokens, capping either changed nothing or bought source coverage by admitting less relevant text — and coverage climbs on its own as the budget grows, from 84% of fetched sources represented at 1,000 tokens to 97% at 4,096. Reach for it when a question wants corroboration rather than depth: two of those six queries had one long page take 86% and 90% of the window, in one case starving the encyclopedia article on the exact term asked about.

Listing pages

front = rustai.extract(html, "https://news.example/")
if front.kind == "index":
    for link in front.links:
        print(link.text, link.url, link.heading_path)
    urls = [l.url for l in front.links]
    articles = client.read(urls)      # now go read them

Scholarly search

client = rustai.Client(
    providers=["arxiv", "openalex", "crossref"],
    contact_email="you@example.com",
)
for hit in client.search("sparse attention long context"):
    print(hit.title)
    print(" ", hit.url)
    print(" ", hit.snippet[:120])   # authors (year). abstract…

Hybrid ranking with your own embeddings

articles = client.read(urls)
units = [u.text for a in articles for u in a.units]     # this exact order

ctx = rustai.slim(
    "왜 위치 인코딩이 필요한가", articles,
    query_vector=model.encode("왜 위치 인코딩이 필요한가").tolist(),
    unit_vectors=model.encode(units).tolist(),
    semantic_weight=0.5,        # 0 is pure BM25, 1 is pure cosine
)

One vector per unit, flattened over articles in order. Short units are dropped before scoring, so the alignment is against every unit rather than the survivors — a wrong-sized list is refused with the expected length rather than quietly mis-ranked. Vectors need not be normalised.

Chunks for a vector store

for chunk in article.chunks(target_tokens=512, overlap_tokens=64):
    store.add(chunk["text"], metadata=chunk)

chunks = rustai.chunk_many(articles)      # parallel across every core

Each chunk is a dict with text, markdown, tokens, url, title, heading_path, index and units — enough for a retrieved chunk to cite itself without a second lookup.

Your own sites, without a search engine

client = rustai.Client(providers=[
    "rss:https://blog.rust-lang.org/feed.xml",
    "sitemap:https://doc.rust-lang.org/sitemap.xml",
])

Feeds and sitemaps are the highest-signal sources here — complete, ordered, and free of anyone else's ranking.

API reference

Every name below is exported from rustai; the stub in python/rustai/_rustai.pyi carries the same signatures with full docstrings, so an editor will complete them.

Entry points

Call Returns Use it when
research(query, **kw) Research One question, one call, throwaway client
Client(**kw) Client Many questions — the connection pool, robots cache and per-host pacing live here
extract(html, url=None, **kw) Article You already have the HTML
extract_many(documents, urls=None, **kw) list[Article] A batch — parallel across every core, GIL released
slim(query, articles, **kw) Context You assembled the articles yourself
chunk_many(articles, **kw) list[dict] Feeding a vector store
research(
    query: str, *,
    max_sources: int = 5,              # pages actually fetched and read
    max_tokens: int = 2048,            # context budget
    max_tokens_per_source: int | None = None,
    providers: Sequence[str] | None = None,
    impersonate: str = "chrome",
    respect_robots: bool = True,
    contact_email: str | None = None,  # OpenAlex/Crossref polite pool
) -> Research
Client(*,
    # search
    providers: Sequence[str] | None = None,
    max_results: int | None = None,    # hits search returns (was `limit`)
    contact_email: str | None = None,
    # fetching
    concurrency: int = 16,
    timeout: float = 20.0,
    impersonate: str = "chrome",       # "chrome", "firefox", "safari", "none"
    respect_robots: bool = True,
    respect_crawl_delay: bool = True,  # arXiv asks 15s; this is why a fetch waits
    per_host_delay: float = 0.25,
    retries: int = 2,
    max_retry_after: float = 60.0,
    max_body_bytes: int = 8 * 1024 * 1024,
    accept_language: str = "en-US,en;q=0.9",
    browser_fallback: bool = False,    # headless Chrome, only after a static fetch fails
    proxies: Sequence[str] | None = None,
    cookie_file: str | None = None,
    # extraction
    include_links: bool = True,
    include_images: bool = False,
    include_tables: bool = True,
    index_mode: IndexMode = "auto",    # "auto" | "never" | "always"
    # compression
    max_tokens: int = 2048,
    max_tokens_per_source: int | None = None,
    diversity: float = 0.35,
)

max_results is search breadth — how many fused hits come back, cheap to raise because nothing is fetched yet. max_sources is read depth — how many of those are actually fetched and extracted, which is what costs time. So Client(max_results=20).research(q, max_sources=5) casts a wide net and reads the best five of it.

Client methods

Method Returns Note
.search(query, *, strict=False) list[SearchResult] Fused, deduplicated. A failing provider degrades the set; strict=True raises only if every one failed
.fetch(urls, *, raise_on_error=False) list[Page] Concurrent. Not 1:1 with input — match on Page.url
.read(urls, *, raise_on_error=False) list[Article] Fetch and extract. Also not 1:1
.research(query, *, max_sources=5) Research All four stages
.save_cookies() None Flush the jar to cookie_file

Objects

Type Fields
Research query, markdown, context, results, articles, failures
Context query, markdown, tokens, selected, sources, units_considered
Article url, kind, title, markdown, text, units, links, meta, stats, tokens, .chunks(...)
Unit kind, level, text, markdown, heading_path, position, tokens
Page url, final_url, status, body, content_type, elapsed_ms, rendered, .extract()
SearchResult url, title, snippet, score, rank, providers
Link url, text, snippet, heading_path
Meta title, description, byline, published, site_name, canonical, language
DenoiseStats html_bytes, markdown_bytes, compression, nodes_visited, nodes_dropped

Article.kind is "article" or "index"; Unit.kind is "heading", "paragraph", "list_item", "code", "quote" or "table". Every object has .to_dict().

Context.selected is one dict per chosen unit — source and unit are indices, into Research.articles and that article's .units:

for sel in ctx.selected:
    unit = articles[sel["source"]].units[sel["unit"]]
    print(sel["relevance"], sel["density"], sel["tokens"], unit.text)

Utilities

Call Returns
count_tokens(text) int — the estimate the budget is spent against
tokenize(text) list[str] — the tokens the ranker uses
density(text) float in [0, 1] — information density
canonical_url(url) str | None — the identity used for deduplication
parse_feed(xml) list[dict] — RSS or Atom
parse_sitemap(xml) dict[str, list[str]] — urls and nested sitemaps

Errors

All inherit RustaiError: NetworkError, HttpStatusError, RobotsError, ExtractError, ProviderError, BrowserError. Batch calls swallow per-item failures by default and report them in Research.failures; pass raise_on_error=True when a missing page is a problem rather than a nuisance.

Rust API

use rustai_core::pipeline::Pipeline;

# async fn run() -> rustai_core::Result<()> {
let pipeline = Pipeline::new()?;
let research = pipeline.research("what is BM25", 5).await;
println!("{}", research.context.markdown);
# Ok(()) }

Every stage is public and usable alone: rustai_core::search::Router, rustai_core::http::Fetcher, rustai_core::parse::extract, rustai_core::rank::slim.

Cargo features

Feature Default What it does
impersonate ✅ Chrome TLS/JA3 + HTTP/2 fingerprint emulation
python — PyO3 bindings (enabled by maturin)
browser — headless-Chrome fallback via chromiumoxide
nightly-simd — SIMD tokenisation inside tl; requires a nightly toolchain

Text scanning in this crate is SIMD-accelerated on stable regardless, via memchr. The nightly-simd feature only affects tl's own HTML tokeniser, which is gated on #![feature(portable_simd)] upstream.

Being a good citizen

This library makes it easy to hit other people's servers quickly, so the defaults lean conservative:

  • robots.txt is honoured for every content fetch, with Crawl-delay respected.
  • Requests to one host are spaced 250 ms apart; total concurrency is capped.
  • Response bodies are capped at 8 MB and aborted mid-stream past that.
  • Retries back off exponentially rather than hammering a rate limit.

Search-provider endpoints you name explicitly (DuckDuckGo's HTML endpoint, a feed URL) skip the robots.txt check, because a query is not a crawl and several of those endpoints disallow the very path they exist to serve. Pages discovered through them go through the normal checked path.

Impersonation exists so ordinary reading is not misclassified as abuse. It is not a licence to ignore a site's terms, and you are responsible for what you point this at.

Be clear-eyed about what it does. Measured against ten sites, the profiles produce genuinely distinct JA3, JA3N and Akamai HTTP/2 fingerprints — but on sites with real bot defences (Cloudflare Enterprise, PerimeterX) turning impersonation on changed the outcome on none of them. Those blocks key on datacenter IP reputation and behaviour, not on the ClientHello. A fingerprint is necessary, not sufficient; proxies is the knob that addresses the rest, and some sites you simply should not be scraping.

What comes back is untrusted

Everything this library returns was written by whoever owns the page. That is the whole point, and it is also the thing to design around.

The output is data, not instructions. result.markdown is built to be pasted into a prompt, and a page is free to fill it with "ignore your previous instructions". No extractor can tell you which sentences were written in good faith, so the defence is structural: put retrieved context behind a delimiter, tell the model in its system prompt that everything inside is reference material rather than direction, and never let a tool call be authorised by something that arrived in it.

prompt = f"<context>\n{result.markdown}\n</context>\n\nQuestion: {question}"

What the library does do is stop a page forging the structure around its own words. Each source is introduced with ## {title} and its URL, so body text opening with one or two hashes is escaped — otherwise a page could write ## Official Python Documentation above a URL it does not own and have it read as a second source. A page's real headings are demoted beneath the source line, and a code block is fenced longer than any backtick run inside it, so it cannot close its own fence and let what follows render as document structure.

Internal addresses are refused. A search result, a SearXNG instance you point at, or a link you feed back in can name 169.254.169.254 or a service bound to the machine running the crawl — and the response would come back extracted into a context window headed for a model. Loopback, private, link-local and .internal targets are rejected, on the first request and on every redirect hop. To crawl your own development server:

client = rustai.Client(allow_private_hosts=True)

A hostname that resolves inward is caught too, at the moment it is resolved: the client's DNS resolver drops internal addresses from every answer, and since the connection is made to the addresses the resolver returned, the address that was vetted is the address dialled. That closes rebinding as well -- there is no second lookup in between to disagree with the first. A name that resolves to a mix keeps its public addresses; one with nowhere public to go fails to resolve.

The headless-Chrome fallback keeps its sandbox. It renders JavaScript from pages you have no reason to trust, so the Chrome sandbox stays on. Set RUSTAI_CHROME_NO_SANDBOX=1 only where the kernel refuses it anyway — root inside a container.

Cookies are credentials. A cookie_file holds session and clearance cookies and is written 0600. Proxy URLs can carry user:pass@; those are masked in Debug output, but they are still in your process and your config.

Benchmarks

Five, because they measure different things and only one of them is about speed. Each is a script in benches/, so every number here is one you can re-run.

Benchmark Question Command
benchmark.py How fast is extraction? python benches/benchmark.py
quality.py Did extraction keep the article? python benches/quality.py /tmp/rustai-pages
retrieval.py Did the answer reach the window? python benches/retrieval.py
tokens.py What did the question cost? python benches/tokens.py
hybrid.py Do embeddings earn their keep? python benches/hybrid.py

Speed

Two corpora, and the difference between them matters more than either number.

Synthetic — 200 generated pages, 46 KB each, deterministic and redistributable. python benches/corpus.py /tmp/rustai-corpus builds it.

Engine Throughput vs trafilatura
rustai.extract_many 5,898 docs/s 39×
rustai.extract (loop) 1,681 docs/s 11×
trafilatura 150 docs/s 1×

Real pages — 15 live pages including Wikipedia articles over 1 MB. Fetch them with benches/fetch_corpus.py.

Engine Serial Across cores
resiliparse 103 docs/s 116 docs/s
rustai 64 docs/s 77 docs/s
justext 14 12
trafilatura 5 4

Real pages are 25× slower per document than synthetic ones, because they are seven times larger and far messier. Quote the second table. The first is useful for spotting a regression against a fixed input, not for deciding whether this is fast enough for you.

trafilatura gains nothing from threads — it holds the GIL. extract_many releases it and spreads across the rayon pool, which is where its multiple comes from. Apple M1, 8 cores, CPython 3.14; notebooks/colab_speed_benchmark.ipynb runs the same comparison wherever you are, and the ranking does move with the hardware.

Extraction quality

python benches/fetch_corpus.py /tmp/rustai-pages
python benches/quality.py      /tmp/rustai-pages --vs-trafilatura

Twenty real pages, scored two ways at once: five-word shingle overlap with each page's own article container, and a count of site furniture that reached the output.

recall precision F1 furniture
rustai 65.3% 85.5% 72.6% 3
trafilatura 68.3% 88.2% 75.1% 4

Both numbers are needed. Judged on overlap alone the best setting is to switch every threshold off — the container being compared against never held the navigation, so passing the whole page through scores better while quadrupling the furniture. That is why min_block_len stays where it is.

What a question costs

python benches/tokens.py

Five stages, each where somebody's pipeline actually stops, over seven questions:

raw HTML     233,184 tokens   answer present 100%
parsed text   65,071  (27.9%)                100%
extracted     18,968   (8.1%)                100%
rustai all    26,058  (11.2%)                100%
rustai slim    1,884   (0.8%)                100%

A dedicated extractor gets you to 8%. Ranking and a token budget get you to 0.8%, a tenth of that, with the answer still present in every question — which is checked, because a benchmark counting only tokens would reward returning nothing. On a 3B model with a 32k window the raw pages behind "딥러닝이 뭐야?" do not fit at all, at 593,439 tokens; the context that answers it is 1,627.

Did the answer arrive?

python benches/retrieval.py

Nine questions spanning explanation, an event on a date, and a current-value lookup. Each carries the patterns any correct source would contain, so the measurement holds no model and no prompt and does not move when you change either: answer present 96%, 90% of the budget spent on sources that mention the subject, median 3.2s.

Adding a caller's embedding model takes the first number to 100% at semantic_weight between 0.25 and 0.5 — python benches/hybrid.py sweeps it.

Caveats worth stating plainly

Single-machine numbers on corpora this repository chose. The extraction ranking against resiliparse has flipped between runs on the same Colab instance. The retrieval and token benchmarks hit the live web, so they move with what search returns that day. Run them on your own pages before trusting any of it.

Development

cargo test                    # Rust unit tests, no network
cargo test --features browser # includes the headless fallback build
maturin develop --release     # build and install into the active venv
pytest                        # Python tests
pytest -m network             # the live-internet tests, off by default

License

Dual-licensed under MIT or Apache-2.0, at your option.