Skip to main content

Crate rustai_core

Crate rustai_core 

Source
Expand description

A zero-cost local research pipeline: collect, denoise and compress the open web into something a small local model can actually read.

No API key, no per-call billing, and no browser-grade DOM in the hot path. The four stages are independent and each is useful on its own:

StageModuleWhat it does
Searchsearchfree providers, queried concurrently, fused by rank
FetchhttpChrome TLS/JA3 fingerprint, robots-aware, capped and paced
Denoiseparse / denoisefind the article, drop the furniture, emit Markdown
SlimrankBM25 + information density + MMR, under a token budget

§The whole pipeline

use rustai_core::pipeline::Pipeline;

let pipeline = Pipeline::new()?;
let research = pipeline.research("what is BM25", 5).await;

println!("{}", research.context.markdown);
for (stage, why) in &research.failures {
    eprintln!("skipped {stage}: {why}");
}

§Just the denoiser

No network, no runtime, no configuration:

let html = r#"
    <html><body>
      <nav><a href="/a">A</a><a href="/b">B</a><a href="/c">C</a></nav>
      <article>
        <h1>Arena parsers</h1>
        <p>Every node lives in one contiguous buffer, so traversal is a
           pointer bump rather than a chase across the heap.</p>
      </article>
      <footer>© 2026. All rights reserved.</footer>
    </body></html>"#;

let article = rustai_core::parse::extract(html, Some("https://example.com/post"))?;

assert_eq!(article.title(), Some("Arena parsers"));
assert!(article.markdown.contains("# Arena parsers"));
assert!(!article.text.contains("All rights reserved"));

§Just the slimmer

Rank blocks you already have against a question, under a token budget that covers the rendered output — headers and breadcrumbs included, not just the blocks themselves:

use rustai_core::rank::{SlimConfig, slim};

let article = rustai_core::parse::extract(
    "<article><h1>Numbers</h1>\
     <p>Streaming 200 documents held 4.8 MiB resident, at 520 documents per second.</p>\
     <p>Paprika is best bloomed in fat before the liquid goes into the pot.</p>\
     </article>",
    Some("https://example.com/post"),
)?;

let context = slim("how much memory", &[article], &SlimConfig::with_budget(64));

assert!(context.markdown.contains("4.8 MiB"));
assert!(context.tokens <= 64);

§Being a good citizen

http::Fetcher honours robots.txt and Crawl-delay, spaces requests to a host, caps concurrency and body size, and backs off exponentially on retry — all by default. Browser impersonation exists so that ordinary reading is not misclassified as abuse; it is not a licence to ignore a site’s terms.

§Cargo features

FeatureDefaultEffect
impersonateyesChrome TLS/JA3 and HTTP/2 fingerprint emulation
pythonnoPyO3 bindings; enabled by maturin
browsernoheadless-Chrome fallback for JS-gated pages
nightly-simdnoSIMD tokenisation in tl; needs a nightly toolchain

Re-exports§

pub use error::Error;
pub use error::Result;

Modules§

chunk
Group units into pieces an embedding model will accept.
denoise
Semantic denoising: find the article, drop everything else.
error
Error type for the whole pipeline.
http
The fetch layer: impersonated HTTP, politeness, and a headless fallback.
parse
HTML → denoised Markdown units.
pipeline
The end-to-end pipeline: a question in, a context window out.
rank
Ranking and compression: what actually goes in the context window.
search
The collection router.
text
Shared tokenisation.