syntext 2.0.0

███████╗██╗ ██╗███╗ ██╗████████╗███████╗██╗ ██╗████████╗ ██╔════╝╚██╗ ██╔╝████╗ ██║╚══██╔══╝██╔════╝╚██╗██╔╝╚══██╔══╝ ███████╗ ╚████╔╝ ██╔██╗ ██║ ██║ █████╗ ╚███╔╝ ██║ ╚════██║ ╚██╔╝ ██║╚██╗██║ ██║ ██╔══╝ ██╔██╗ ██║ ███████║ ██║ ██║ ╚████║ ██║ ███████╗██╔╝ ██╗ ██║ ╚══════╝ ╚═╝ ╚═╝ ╚═══╝ ╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═╝

A faster grep for agentic AI. Up to 20x+ faster than ripgrep on LARGE codebases.

Hybrid code search index for agent workflows, built in Rust. Indexes repositories using sparse n-grams, then narrows to a small candidate set before verification. Drop-in replacement for rg in AI agent loops where grep is called repeatedly and in parallel.

Status: stable (v2.0).

Installation

Quick install (macOS and Linux)

curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh

Installs st to /usr/local/bin. On macOS, uses Homebrew cask if brew is available. On Debian/Ubuntu (x86_64), installs the .deb package. All other Linux targets get the raw binary. Checksums are verified against SHA256SUMS from the release.

Override defaults with environment variables:

INSTALL_DIR=~/.local/bin SYNTEXT_VERSION=2.0.0 \
  curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh

brew tap whit3rabbit/tap
brew install --cask whit3rabbit/tap/syntext

VERSION=2.0.0

# Debian/Ubuntu (x86_64)
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/syntext_${VERSION}_amd64.deb" \
  -o "syntext_${VERSION}_amd64.deb"
sudo dpkg -i "syntext_${VERSION}_amd64.deb"

# Any Linux (x86_64 or arm64)
ARCH=amd64   # or arm64
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/st-${VERSION}-linux-${ARCH}" -o st
chmod +x st && sudo mv st /usr/local/bin/

iwr -useb https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.ps1 | iex

Installs st.exe to %LOCALAPPDATA%\syntext and adds it to the user PATH. Restart your terminal after install.

To pin a version or run from a saved script:

powershell -ExecutionPolicy Bypass -File install.ps1

Prebuilt WASM packages are available on the releases page as syntext-wasm-<version>.tar.gz. To build from source:

cargo install wasm-pack
wasm-pack build --target bundler -- --features wasm --no-default-features
# output: pkg/  (JS glue + .wasm + TypeScript types)

Other targets: --target nodejs, --target web.

From source

cargo install syntext

Benchmarks

Search latency across five real-world repositories (v2.0, macOS, Apple Silicon).

Average speedup across five presets: 1.7x versus rg. Search time excludes index build time.

[!NOTE] Speedup is most significant on large repositories and selective queries where the index eliminates the need to scan tens of thousands of files. Performance is also substantially faster on Linux than macOS; Linux utilizes kernel-level openat2(RESOLVE_BENEATH) for secure path containment, completely bypassing the user-space canonicalization and metadata check overhead required on macOS.

See docs/BENCHMARKS.md for methodology, index build times, query discipline, and historical runs.

Usage

# Build the index (run once per repo, then only after large changes)
# Index is stored in .syntext/ at the repo root (nearest .git ancestor).
# Not run automatically -- you must run this before the first search.
st index
st index --stats                    # show file count and index size after build

# Override where the index is stored or which root to index
st --repo-root /path/to/repo index
st --index-dir /tmp/my-index index

# After editing files, sync the index incrementally (faster than full rebuild)
st update

# Search the whole repo (index must exist)
st "fn parse_query"                 # regex
st -F "parse_query("                # literal (metacharacters stay literal)
st -i "parsequery"                  # case-insensitive
st -x "TODO"                        # whole-line match
st -n "impl.*Iterator"              # force line numbers

# Restrict search scope with positional paths
st "needle" src/                    # search one directory
st "needle" src/lib.rs              # search one file
st "needle" src/lib.rs tests/       # search multiple files/directories

# Additional filters and output modes
st -t rs "impl.*Iterator"           # restrict to Rust files
st -g "src/" "TODO"                 # restrict by glob
st -c "parse_query" src/lib.rs      # count matches in one file
st -l "parse_query"                 # print matching file paths
st --json "TODO"                    # NDJSON output for tooling

# Status
st status

Notes:

• Search is the default command, there is no st search subcommand.
• Like ripgrep, file names are shown by default when searching a directory, the whole repo, or multiple positional paths.
• Like ripgrep, line numbers are off by default when stdout is not a TTY. Use -n to force them on.

Fallback to ripgrep/grep (un-indexed search)

By default, searching a path with no index fails with exit code 2 and tells you to run st index. For agent harnesses that sometimes search outside an indexed checkout (e.g. a throwaway clone in /tmp), st can instead fall back to ripgrep (preferred) or grep so the search still returns results.

It is opt-in. Enable it with the --fallback flag or SYNTEXT_FALLBACK_RG=1 (accepts 1, true, yes, on):

SYNTEXT_FALLBACK_RG=1 st "needle" /tmp/some-clone   # env var (set once in harness)
st --fallback "needle" /tmp/some-clone              # per-invocation flag

Behavior:

• Triggers only when the index is missing. A corrupt index or lock conflict still fails loudly so real problems are never masked.
• ripgrep receives your original arguments unchanged (st's CLI is a superset of rg's), so --json, --vimgrep, context, and filter flags produce exactly the output you would get from rg directly.
• grep is the last resort when rg is not on PATH. It is best-effort: common match flags are mapped, but output-only modes that grep cannot produce (--json, --vimgrep, --heading, --column, -t/--type) are dropped.
• The fallback is slower than the index and prints a one-line notice to stderr (suppressed under --quiet); stdout is left clean for parsing.

This is a convenience for un-indexed paths, not a replacement for st index: build an index for full speed and syntext's coverage guarantees.

Agent harness install

st can install RTK-style agent harness integrations. Programmatic hooks rewrite safe agent shell searches from rg or grep to st only when a .syntext/ index exists. Human shells, scripts, pipes, CI, and unsupported search forms are left alone. Hooks never run st index or st update automatically.

Quick installs:

# Claude Code project instructions only
st init

# Claude Code global Bash hook plus Grep blocker
st init -g

# RTK-style agent selectors
st init -g --agent cursor
st init -g --gemini
st init --copilot        # project hook; `st init -g --copilot` is also accepted
st init --codex          # project rules
st init -g --codex       # global Codex rules

Explicit install, show, and uninstall commands are also available:

st agent install claude --global
st agent show claude --global
st agent uninstall claude --global

Supported harnesses:

Each install is idempotent, preserves unrelated settings, writes a timestamped backup before editing an existing file, and only removes syntext-owned entries on uninstall.

Architecture

Query -> Router -> [Literal | Indexed Regex | Full Scan]
                        |
                   Gram extraction
                        |
                   Posting list intersection (smallest-first)
                        |
                   Candidate file IDs
                        |
                   Verifier (memchr or regex against file content)
                        |
                   Results

Three index components:

• Content index: sparse n-gram posting lists. Context-independent forced boundaries ensure no false negatives for token-aligned queries.
• Path index: Roaring bitmap component sets for path/type filtering.
• Symbol index (optional): Tree-sitter extraction into SQLite.

Segments are immutable single-file mmap structures (SNTX format). Updates commit atomically to an in-memory overlay via ArcSwap, while durable incremental HEAD-move updates are written as LSM-style delta segments with a checksummed delete-set sidecar.

See docs/ARCHITECTURE.md for the full quantitative analysis: selectivity math, index size estimates, posting list encoding tradeoffs.

WASM

The wasm Cargo feature compiles syntext to a fully in-memory index with no filesystem access. See the releases page for prebuilt syntext-wasm-<version>.tar.gz, or build from source:

wasm-pack build --target bundler -- --features wasm --no-default-features
# output: pkg/  (JS glue + .wasm + TypeScript types)

Known limitations

1. Crash recovery: Uncommitted in-memory overlay state (used by resident integrations) is lost on unclean shutdown. For CLI searches, index state is persisted to disk via delta segments and delete sidecars, and any staleness is auto-healed on the next search via automatic bounded update-on-search. If a sidecar is corrupted, the index fails closed and requires a re-index or update.
2. Non-aligned substring coverage: ~16% false-negative rate for queries that don't align with token boundaries. Token-aligned queries (identifiers, keywords) have 0% false negatives.
3. Network filesystems: Index directory must be on local filesystem. NFS/SMB behavior is undefined.
4. Case-insensitive overhead: ~15-20% more candidates due to lowercase normalization. Correct results are guaranteed by the verifier.
5. \r-only line endings: Treated as a single line (matches ripgrep behavior).
6. Symbol search accuracy: Tier 3 (heuristic) results are approximate. Tree-sitter failures fall back silently.
7. One root per index: Each index covers exactly one --repo-root. There is no way to merge multiple directories into a single index. To search across two repos, build and query each index separately with --repo-root. st update requires a git repo; non-git directories must be re-indexed with st index.

Design documents

• docs/ARCHITECTURE.md -- Quantitative analysis: selectivity math, index size estimates, posting list encoding, design tradeoffs

License

MIT