basemind 0.24.0

---
title: Document Search
description: Semantic and full-text search over PDFs, Office files, HTML, email, images, and web pages — with OCR, reranking, and NER filtering.
---

import { Aside, Badge, Card, CardGrid } from '@astrojs/starlight/components';

basemind extracts 90+ file formats (PDF, Office, HTML, email, images via OCR) into a
LanceDB vector store and answers meaning-based queries with optional cross-encoder reranking.
Web pages scraped or crawled into the same store are searchable the same way.

<Badge text="--features documents" /> or `--features full`

## Requirements

- The `memory` MCP tool's `documents` mode needs a build with `--features documents` (or
  `full`). The other `memory` modes (`put`, `get`, `list`, `search`, `delete`, `audit`) are
  gated on the separate `--features memory` (also in `full`). Without the relevant feature
  the tool is advertised but the mode's body returns an error.
- Web ingestion (`web` modes `scrape`, `crawl`, `map`) needs `--features crawl`. When that
  feature is off the `web` tool is not registered at all.
- Documents must be scanned first: `basemind scan` with the documents feature extracts
  and embeds them into `.basemind/`.

## Core tool

### `memory` mode `documents`

Semantic search across PDFs, Office, HTML, email, images (OCR), and web pages. Returns
chunk-level hits with path, matched text, byte span, vector distance, and — when enabled
at scan time — cross-encoder rerank score, named entities, and document summary. CLI:
`basemind memory documents`.

```json
{
  "mode": "documents",
  "query": "how is the index schema versioned",
  "limit": 10,
  "entity_category": "PERSON",
  "keywords_contains": "release"
}
```

Filter results by:

- **`entity_category`** — narrow to documents mentioning entities like `PERSON`, `LOCATION`,
  `ORGANIZATION` (NER-extracted at scan time).
- **`keywords_contains`** — narrow to documents with a matching keyword (extracted at scan
  time).
- **`mime_type`** — filter by file type, e.g. `"application/pdf"` or `"text/html"`.

Each hit carries:

- `path` — file path or web scope (`web:<host>/<path>`).
- `chunk_idx` — which chunk within the document.
- `text` — the matched passage.
- `byte_span` — precise location in the source.
- `distance` — L2 vector distance (lower = better).
- `rerank_score` — cross-encoder score in `[0, 1]` (higher = better, only when reranking
  is enabled).
- `keywords` / `entities` / `summary` — document-level metadata (when enabled at scan).

## Web ingestion

### `web` mode `scrape`

Fetch and index a single page, adding it to the document store. CLI: `basemind web scrape`.

```json
{
  "mode": "scrape",
  "url": "https://docs.example.com/guide",
  "scope": "docs"
}
```

Results land tagged with a `scope` (default: `web:<host>`), making them queryable the same
way as local documents.

### `web` mode `crawl`

Follow links from a seed URL up to a configurable depth, indexing all discovered pages.
CLI: `basemind web crawl`.

```json
{
  "mode": "crawl",
  "url": "https://docs.example.com/",
  "max_depth": 2,
  "max_pages": 100
}
```

<Aside type="note">
The crawler honors `robots.txt` by default (disable with config). It SSRF-blocks private
and loopback hosts unless `[crawl].allow_private_network = true` is set in
`.basemind/basemind.toml`.
</Aside>

### `web` mode `map`

Discover a site's pages without fetching bodies — returns a sitemap. CLI: `basemind web
map`.

```json
{
  "mode": "map",
  "url": "https://docs.example.com"
}
```

Useful for "what pages exist on this site?" without the download cost.

## Shared memory

Store key–value pairs that persist across sessions and are queryable by any agent on the
same repo (determined by normalized git origin URL). Unrelated repos keep separate memory.

### `memory` mode `put`

Store a value. CLI: `basemind memory put`.

```json
{
  "mode": "put",
  "key": "architecture_notes",
  "value": "The scanner uses rayon parallel iteration..."
}
```

### `memory` mode `get` / `list`

Retrieve a value by exact key or list keys by prefix. CLI: `basemind memory get` /
`basemind memory list`.

```json
{
  "mode": "get",
  "key": "architecture"
}
```

### `memory` mode `search`

Semantic search over stored values. CLI: `basemind memory search`.

```json
{
  "mode": "search",
  "query": "how is parallelism managed",
  "limit": 5
}
```

## Configuration

In `.basemind/basemind.toml`:

```toml
[documents]
enabled = true
embed = true  # set to false to disable embeddings (keyword search only)

[documents.reranker]
enabled = true  # cross-encoder reranking for document chunks
```

## Discipline

- **Use `memory` mode `documents` instead of opening PDFs/Office/HTML by hand.** Semantic
  search finds passages about a topic without keyword matching.
- **Use `web` modes `scrape` / `crawl` to pull docs into the RAG store.** Once indexed, they
  are searchable alongside local documents.
- **Use `memory` modes `put` / `search` for agent-generated insights.** Store findings that
  you want to recall in future sessions.
- **Filter by `entity_category` or `mime_type` when the query is too broad.** Narrow results
  to a document class.

<Aside type="note">
All results are paginated. Use `next_cursor` from one response as `cursor` in the next to
fetch additional pages.
</Aside>

## See also

[Code intelligence](/capabilities/code-intelligence/) · [Git intelligence](/capabilities/git-intelligence/) ·
[Code search](/capabilities/code-search/)