Crate autumn_search

Expand description

Keyword and vector search for autumn-web applications (issue #1191).

Autumn already had full-text search primitives in core (#842): a #[searchable] model attribute, a tsvector column, and a #[repository(searchable)] search() method. What it did not have was a search subsystem — a way to declare a model searchable, keep an index in sync as records change, and query it through one engine-agnostic API, let alone semantic / vector search for “find similar” and RAG retrieval.

This crate is that subsystem: the autumn-shaped answer to Laravel Scout, with vectors first-class rather than bolted on.

§Quick start

use std::sync::Arc;
use autumn_search::{SearchPlugin, SearchSyncHooks};

// 1. Mark the model searchable. `embed` nominates the field to embed.
#[autumn_web::model]
#[searchable(language = "english")]
pub struct Article {
    #[id] pub id: i64,
    #[searchable(weight = "A")] pub title: String,
    #[searchable(weight = "B", embed)] pub body: String,
}

// 2. Keep the index in sync from the record lifecycle. `#[repository]`
//    takes a plain type NAME for `hooks`, so alias the generic first.
type ArticleSearchHooks = SearchSyncHooks<Article, NewArticle, UpdateArticle>;

#[autumn_web::repository(Article, hooks = ArticleSearchHooks, commit_hooks = true)]
pub trait ArticleRepository {}

// 3. Mount the plugin.
autumn_web::app()
    .plugin(
        SearchPlugin::new()
            .postgres()
            .embedder(Arc::new(MyEmbedder))
            .index::<Article>(),
    )
    .run()
    .await;

// 4. Query it. The plugin installs the client as an `AppState` extension.
let search = state.extension::<SearchClient>().expect("SearchPlugin installed");
let page = search.search::<Article>("rust web framework", &page_req).await?;
let similar = search.similar::<Article>("how do I add auth?", 5).await?;

Configuration comes from [search], which the plugin resolves itself (base autumn.toml → [profile.<name>.search] → autumn-<profile>.toml → AUTUMN_SEARCH__*, the same layering core applies) unless the app passes SearchPlugin::config(...).

§How the pieces fit

Concern	Type	Notes
Index shape	`autumn_web::search::IndexDefinition`	derived from `#[searchable]`, never hand-written
Record extraction	`autumn_web::search::SearchDocument`	derived from `#[searchable]`
Engine	`SearchBackend`	`MemorySearchBackend`, `PostgresSearchStore`, or yours
Embeddings	`Embedder`	no model or vendor ships here, by design
Index sync	`SearchSyncHooks` → `ReindexArgs` → `#[job]`	off-request and durable
Reading records	`DocumentSource`	drives both reindex and backfill
Authorization	`SearchVisibility` → `SearchFilter`	pushed into the engine query
Querying	`SearchClient`	`Page`/`ListQuery`-shaped results

§Design notes worth knowing

Reindex re-reads the row. A reindex job carries (index, id), not the record. So create, update, delete, a replayed job, a lost event, and a row changed by direct SQL all converge to the same index state, and at-least-once delivery is safe by construction.

Filters are arguments, not afterthoughts. SearchFilter is a required parameter of the backend query methods, so an engine cannot quietly return rows the caller may not see; filters intersect, so a caller can only ever narrow what authorization allowed.

Query text is a bag of words. Every token must match, and operator syntax (OR, field:, *) is never interpreted. That keeps results consistent across engines and makes a hostile query structurally incapable of widening the result set.

Structs§

BackendCapabilities: What a backend can do, so callers can degrade deliberately.
BackfillArgs: Payload of the BACKFILL_JOB.
BackfillOptions: Knobs for a full reindex.
BackfillReport: What a backfill did.
DenyAll: A SearchVisibility that denies everything.
HashingEmbedder: A deterministic, dependency-free hashing vectorizer.
IndexDefinition: The engine-agnostic description of one model’s search index.
IndexedDocument: A SearchDocument paired with its (optional) embedding.
KeywordQuery: A ranked keyword query.
ListQuery: Sort/filter parameters parsed from the query string, composing with PageRequest to drive an allowlisted list query.
MemoryDocumentSource: An in-memory DocumentSource for tests and for apps that already hold their corpus in process.
MemorySearchBackend: An in-memory SearchBackend.
NoEmbedder: The default embedder: refuses to produce vectors.
Page: Paginated response wrapper with navigation metadata.
PageRequest: Pagination parameters parsed from the query string.
PostgresSearchStore: Postgres-backed search storage.
ReindexArgs: Payload of the REINDEX_JOB.
SearchClient: The application-facing search API.
SearchClientBuilder: Builder for SearchClient.
SearchConfig: Configuration for the search subsystem.
SearchDocument: A single record, extracted into the shape every backend indexes.
SearchFieldValue: One extracted field value of a SearchDocument.
SearchFilter: Restrictions applied inside the backend query.
SearchHit: One ranked result.
SearchIndexField: One indexed field of a searchable model.
SearchPlugin: Keyword + vector search for autumn-web applications.
SearchSyncHooks: MutationHooks that keep a searchable model’s index in sync.
TenantVisibility: Restricts every search to the ambient tenant.
VectorQuery: A vector / k-NN similarity query.

Enums§

BackfillTarget: What a one-shot backfill boot was asked to rebuild.
ReindexOp: What a reindex instruction should do.
SearchConfigError: Why a [search] section was rejected.
SearchError: Everything that can go wrong indexing or querying.
VectorMode: How embeddings are physically stored.

Constants§

BACKFILL_ENV: Environment variable that turns a normal boot into a one-shot backfill.
BACKFILL_JOB: Job name for the full backfill.
BACKFILL_PURGE_ENV: Environment variable that makes the one-shot backfill purge first.
DOCUMENTS_TABLE: Physical table every index’s documents live in.
REINDEX_JOB: Job name for the per-record reindex. A wire contract: in-flight payloads outlive a deploy, so this string must not change.
TENANT_FILTER_KEY: The reserved SearchFilter::equals key that maps onto SearchDocument::tenant_id rather than an indexed field.

Traits§

DocumentSource: Reads records out of the system of record and extracts them into SearchDocuments.
Embedder: Turns text into vectors for semantic search.
SearchBackend: A search engine autumn-search can drive.
SearchIndexed: A model that #[model] #[searchable] has given a search index.
SearchTextValue: Converts a model field into the text a search backend indexes.
SearchVisibility: Turns a request context into the restriction a search must run under.

Functions§

ambient_tenant_filter: The ambient-tenant restriction for definition.
backfill_request_from_env: The backfill requested through the environment, if any.
client_from_state: Pull the installed SearchClient off application state.
cosine_similarity: Cosine similarity of two vectors, in [-1.0, 1.0].
current_tenant_filter: The ambient-tenant restriction, or an empty filter outside a tenant scope.
empty_page: Build an empty page for request — the fail-closed result every backend returns for a blank query or an impossible filter.
enqueue_reindex: Enqueue a reindex for one record.
enqueue_reindex_for: Enqueue an upsert reindex for record.
enqueue_unindex_for: Enqueue a delete reindex for record.
normalize: Scale vector to unit length in place. A zero vector is left alone (it has no direction), which keeps cosine_similarity free of NaN.
query_tokens: Tokenize a user query, returning None when nothing usable remains.
search_job_infos: The JobInfo set for the search jobs, routed to queue.
tokenize: Split text into lowercase alphanumeric tokens.

Type Aliases§

BoxFuture: Boxed future returned by the object-safe search traits.
SearchResult: Result alias used throughout autumn-search.