Crate link_r 

link-r — a tiny, embeddable crawl-and-resolve link index.

Give it a parent HTTPS link. It recursively crawls sublinks to a configurable depth, distills keywords/topics from each page, and builds a small, mmap-able hybrid index (dense embeddings + BM25 + structured metadata). A query resolves to ranked remote URLs — document bodies stay at the URL and are fetched lazily by the caller. Each link is unique in the index (canonical-URL keyed), so re-crawling never duplicates a link.

Design tenets

• Zero-copy: the on-disk index is the file; search runs against mapped pages with no deserialization step.
• Static dispatch: async traits use GATs (no Box<dyn Future>); runtime variant selection uses enum-dispatch, not dyn.
• Self-contained: no server, no sidecar database; the production embedder (OnnxEmbedder, feature onnx) and a deterministic HashEmbedder fallback both run in-process.

Dead-simple API

use link_r::prelude::*;

let mut idx = LinkIndex::open_or_create("kb.lnkr")?;
idx.update("https://example.com/docs/").depth(2).run().await?;
idx.save()?;

for hit in idx.search("how does auth work", 10).await? {
    println!("{:.3}  {}", hit.score, hit.url);
}

// Maintain the knowledge base: re-validate links older than a day, evict dead ones.
idx.refresh().ttl(std::time::Duration::from_secs(86_400)).run().await?;

Re-exports

pub use auth::AnonymousAuth;

pub use auth::AuthProvider;

pub use auth::Credential;

pub use auth::DynAuthProvider;

pub use auth::StaticTokenAuth;

pub use embed::Embedder;

pub use embed::EmbedderId;

pub use embed::HashEmbedder;

pub use error::Error;

pub use error::Result;

pub use extract::AutoExtractor;

pub use extract::Descriptor;

pub use extract::Extractor;

pub use fetch::FetchMeta;

pub use fetch::FetchOptions;

pub use fetch::Fetched;

pub use fetch::Fetcher;

pub use index::DocMeta;

pub use index::Document;

pub use index::Filter;

pub use index::Index;

pub use index::IndexBuilder;

pub use index::UpsertOutcome;

pub use metric::Metric;

pub use payload::ByteStream;

pub use payload::DocPayload;

pub use payload::MmapView;

pub use query::Fusion;

pub use query::Hit;

pub use query::Hits;

pub use query::PreparedQuery;

pub use query::RankParams;

pub use resource::DocId;

pub use resource::Page;

pub use resource::Resource;

pub use resource::ResourceKind;

pub use resource::SourceRef;

pub use source::Source;

pub use url_key::canonicalize;

pub use url_key::UrlKeyExt;

pub use facade::GithubUpdateBuilder;

pub use facade::LinkIndex;

pub use facade::SearchResult;

pub use facade::UpdateReport;

pub use fetch::HttpFetcher;

pub use source::FsSource;

pub use source::CrawlConfig;

pub use source::CrawlScope;

pub use source::CrawlSource;

pub use source::GitHubSource;

pub use source::GithubAuth;

pub use source::GithubSpec;

Modules

auth

Consumer-injected authentication.

bridge

The graph-r bridge (feature graph): absorb a link-r index into the persistent graph, keep it fresh from crawl/refresh reports, and drive link-r’s refresh loop from graph-r due-lists.

embed

Embedders: turn distilled text into dense vectors.

error

Error and result types for link-r.

extract

Extraction: distill a fetched document into the small descriptor the index stores.

facade

LinkIndex — the dead-simple, three-verb API over the trait layer, plus the knowledge-base lifecycle (persist, refresh after a TTL, retention).

fetch

Fetching: retrieve one resource’s bytes.

index

The hybrid index: on-disk format, zero-copy store, retrieval, and incremental upsert.

kb

One knowledge base (feature kb) from the three libraries, on one object store:

metric

The vector distance metric, shared by the embedder and the index.

payload

Zero-copy document payloads.

prelude

Everything you need for the dead-simple API, in one glob import.

query

Query inputs and search results.

remote

Object storage for indexes (feature remote).

resource

Resource addressing: what a source discovers and what gets indexed.

source

Sources: discover the pages to index from a parent.

text

Text primitives: graph-r’s text rules (tokenization, case folding, char n-grams, stopwords), so the hash embedder, BM25, keyword extraction and graph-r’s lexicon all agree on what a term is.

url_key

URL canonicalization and the UrlKey dedup/uniqueness key.

Structs

Url

A parsed URL record.

UrlKey

Durable identity of a document node: xxh3 of its canonical URL.