holys3

grep for S3. Indexed regex search over buckets — fetches matches, not corpora.

holys3 index s3://my-logs/prod                     # build the index, in-bucket, once
holys3 'req-7f3e9a2c1b' s3://my-logs/prod          # then grep it in ~a second
holys3 -i 'timeout' s3://my-logs -g '*.gz' -C2 --since 6h

S3 has no native grep. The alternatives scan: download-everything-and-rg pays for every object on every query, and Athena bills per byte scanned. holys3 builds a compact trigram index next to the data (under .holys3/ in the same bucket), uses it to narrow each query to candidate objects, then fetches only those and verifies with real Rust regexes against the original bytes. The index narrows candidates; the verifier decides matches — results are always exact, never index-approximated.

In the tracked real-S3 benchmark below, a literal query over 1,000 objects finished in 1.5 s at concurrency 64 versus 54.7 s sequentially. A no-match query fetched zero objects. Every benchmark corpus, planted hit count, candidate count, and byte count is deterministic and checked before timing.

Install

Prebuilt binaries for Linux (x86_64, arm64), macOS (Intel, Apple Silicon), and Windows ship with every GitHub release:

cargo binstall holys3   # fetches the prebuilt binary for your platform
cargo install holys3    # or build from source (Rust 1.88+)

Release archives include SHA-256 checksums and GitHub build-provenance attestations. Verify an archive with gh attestation verify -R TalkingComputers/holys3 <archive>.

Quickstart

# S3, with an `aws sso login` session — holys3 reads SSO profiles directly
AWS_PROFILE=my-sso holys3 index s3://my-log-bucket/prod --region us-east-2
AWS_PROFILE=my-sso holys3 'level":"ERROR' s3://my-log-bucket/prod --region us-east-2

# rg-style flags work
holys3 'req-[0-9a-f]+' s3://my-log-bucket --json | jq .
holys3 -w -F 'foo(' s3://my-code-bucket -l

# local directories work too
holys3 index ./logs --out holys3.idxdir
holys3 'TODO|FIXME' ./logs --index holys3.idxdir

The CLI follows ripgrep: holys3 PATTERN TARGET, where TARGET is s3://bucket[/prefix] or a local path. To search for a pattern named like a subcommand, use -e: holys3 -e index s3://bucket.

What's supported

Object formats

Format detection is magic-first. Brotli and zlib have no reliable container magic, so only .br, .zlib, and .zz select those decoders, and the entire stream must validate. Other extensions are never trusted.

Projection and decompression happen at one canonical decoder boundary, so the index and verifier see identical bytes. Columnar line numbers refer to rows. Archive member paths are normalized without filesystem extraction; unsafe paths fail the source, and duplicate normalized names receive #2, #3, etc.

Truncated or corrupt-tailed streams salvage: the cleanly decoded prefix is searched and a warning names the object. Undecodable objects are excluded loudly, never silently searched incorrectly.

Search flags (ripgrep semantics)

-e PATTERN        multiple patterns, OR              -n / -N      line numbers on/off
-F                fixed strings                      --column     1-based match column
-i / -S / -s      ignore / smart / sensitive case    --heading    group under key (tty default)
-w                word boundaries (rg half-bounds)   --no-heading key:line:text (pipe default)
-l                files with matches                 -g GLOB      include/!exclude key globs
-c                count matching lines               -q           quiet, exit at first match
--count-matches   count individual matches           --color WHEN auto/always/never/ansi
-m NUM            max matching lines per object      --json       rg-compatible JSON Lines
-A/-B/-C NUM      context lines with -/-- separators --stats      candidate stats to stderr

Exit codes are rg's: 0 match found, 1 no match, 2 error. Patterns are line-oriented like rg: ^/$ anchor at every line, character classes never match the line terminator, and a literal \n in a pattern is an error.

holys3-specific scoping

• --key-prefix P — only keys starting with P (prunes whole index segments before any fetch)
• --key-regex RE — only keys matching RE
• --since T / --until T — only objects whose key-embedded timestamp overlaps the window. T is 2026-06-09, 2026-06-09T14:30[:00][Z], or relative 30s/15m/6h/2d/1w (ago, UTC). Recognized key shapes: 2026/06/09 paths, year=2026/month=06/day=09[/hour=14] hive partitions, dt=/date=2026-06-09, ALB/CloudTrail filename stamps (20260609T2300Z), and CloudFront/S3-access-log dashed stamps (2026-06-09-23[-59-59]). Keys without a recognizable timestamp are searched anyway, with a note on stderr — time scoping never silently hides data.
• --region, --endpoint (MinIO, R2, any S3-compatible store), --concurrency (default 750), --index (local index dir)
• --object-cache DIR --object-cache-cap BYTES — explicit private cache for immutable S3 source bodies. Both flags are required; files are checksummed, content-addressed by endpoint/bucket/key/ETag, owner-only, atomically replaced, and size-bounded with FIFO eviction.

Credentials

Resolved in order: AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (+ optional AWS_SESSION_TOKEN) from the environment → the AWS_PROFILE config profile, including AWS IAM Identity Center (SSO) profiles whose cached aws sso login tokens are exchanged for role credentials and auto-refreshed before expiry → static profile keys. Requests are signed by holys3's own SigV4 implementation, tested against AWS signature vectors.

How it works

1. The query planner extracts gram constraints from the regex — prefix, suffix, and required inner literals (Cox-style), so .*ERROR.* prunes instead of scanning.
2. The term dictionary (an FST, adaptively prefix-sharded for dense trigram spaces) maps each gram to its postings offset and doc count, so selectivity is known before any fetch: absent grams answer instantly, only the rarest grams per AND-group are fetched, and grams present in every doc cost zero bytes on disk and zero fetches.
3. Posting blocks are read with coalesced ranged GETs; candidates are pruned by key scope before a single object is fetched.
4. Candidate physical sources stream through concurrent conditional GETs (If-Match against the indexed ETag), with adaptive AIMD concurrency, retries, request hedging, and a 512 MiB in-flight byte budget. Sources of at least 64 MiB use four concurrent streamed 8 MiB conditional ranges. One archive is fetched and decoded once even when many members are candidates. Sources are then regex-verified on a worker pool — results print as they complete, unordered across objects like rg's parallel mode, pipe-friendly (holys3 ... | head terminates cleanly).

The index

holys3 index s3://bucket/prefix maintains content-addressed segments under .holys3/. Runs are incremental diffs: only new or changed objects are fetched and indexed, deletions take effect immediately, an unchanged bucket costs one listing (~seconds), and small segments merge automatically. Posting lists are density-classed and bit-packed, and grams shared by every document cost nothing. Index construction uses bounded sorted runs instead of a global in-memory postings map. Replaced segments are garbage-collected; the root pointer swap is a compare-and-swap (S3 conditional writes), so a racing concurrent index run fails loudly instead of corrupting anything. Large index blobs upload as concurrent multipart parts. Every immutable segment blob has its own SHA-256 length/hash contract; readers reject truncation, corruption, duplicate segment IDs, and malformed metadata before using it.

Local directories use the same format-8 physical-source/logical-document tables, written to --out (default holys3.idxdir) with BLAKE3 content freshness tokens. Local verification rechecks the token, local runs are incremental, and --rebuild re-ingests everything. Raw candidate files are read concurrently under the same 512 MiB byte budget; expanding formats stay serial so decompression cannot multiply peak memory.

Performance

Real-world S3 — 1,000 synthetic 4 KiB objects, indexed search (concurrency 64) vs a sequential (--concurrency 1) baseline, laptop → us-east-2 over the public internet (per-object RTT dominates; in-region is far lower). Reproduce with make bench-s3.

Absolute latencies vary with the network (per-object RTT dominates from a laptop; in-region EC2 is far lower) — the prune ratios and hits are the stable part.

Continuous (CI) — measured on every push against a pinned local MinIO image (make bench-minio). CI runs release binaries, rejects missing or unbaselined microbenchmarks and hybrid sort paths more than 15% slower than their same-run controls, validates exact end-to-end hit counts, indexes 25,000 objects, and enforces hosted-run time plus a 300 MiB peak-RSS ceiling for high-cardinality and 256 MiB decoded workloads under both index strategies. Segment construction also enforces its cap on logical documents, including archive members, rather than only on physical source objects. A separate 512 MiB compressed-expansion gate caps trigram/sparse index and files-only search RSS at 96 MiB. A 512 MiB raw-object MinIO gate applies the same ceiling to S3 indexing and files-only search with both strategies. A separate 64 MiB high-entropy gzip gate exercises ranged source download, streaming decode, bitmap-backed trigram construction, cold term-cache population, and mmap lookup under the same 96 MiB ceiling. Workspace line coverage is gated at 80%.

Microbenchmarks: make bench-micro. PR CI compares the base and head revisions on one runner and gates statistically confident regressions above 20%; the committed benches/baseline.json remains the reporting reference. Refresh it only from CI's bench-micro artifact.

Limitations

• Raw (unframed) snappy has no magic bytes and is undetectable by design — unsupported as an object format (it still decodes fine inside Avro files, where the container names the codec). lz4 legacy frames (lz4 -l output) are detected and rejected loudly rather than decoded.
• No multiline mode: patterns that would match across line boundaries are line-restricted exactly like rg without -U.
• Decoded output above 8 MiB spills to private temporary files. Trigram indexing and files-only bounded regex verification stream those files; sparse indexing reads them through a bounded 1 MiB window. Expansion is capped at 64 GiB per physical source, 100,000 regular archive members, and four nested format layers.
• Oversized local sources, S3 sources of at least 64 MiB, and large optional object-cache entries remain file-backed through indexing and verification instead of materializing the full body in memory. File-backed gzip, zstd, bzip2, Snappy, Brotli, and zlib sources decode through bounded readers rather than whole-source mappings.
• A search that races segment garbage collection reopens the new index root once before emitting results. Continuous write churn can still require a manual rerun; no incorrect matches are returned.
• Concurrent holys3 index runs over one prefix are safe (the loser errors cleanly and retries), but the design assumes occasional writers, not a write-heavy pipeline.
• AWS general-purpose buckets and S3-compatible endpoints are supported. S3 Express directory buckets require zonal endpoints and session credentials and are not yet supported.
• The library crates are not a stable API; the CLI is the supported surface.

Security

Use private buckets. The index lives in the same account and bucket namespace as the data, under .holys3/; holys3 talks only to your S3 endpoint and never sends data anywhere else.

Contributing

Read ARCHITECTURE.md before changing index, query, S3, or SigV4 behavior. The differential test suites are the correctness contract: indexed search must exactly equal a decoded full scan, for every format, both gram strategies, and every index lifecycle state.

License

Licensed under either of:

• MIT license (LICENSE-MIT)
• Apache License, Version 2.0 (LICENSE-APACHE)