Struct ArtifactStore 

pub struct ArtifactStore<P: ArtifactPolicy> { /* private fields */ }

The local content-addressed store.

Implementations

impl<P: ArtifactPolicy> ArtifactStore<P>

pub fn open(config: impl Into<Config>) -> Result<Self>

pub fn open_read_only(config: impl Into<Config>) -> Result<Self>

Open another process’s store for lookups and restores on a read-only mount, such as a CI container given the host’s store through a :ro volume. The mount must enforce filesystem protection: SQLite can create WAL side files on a writable mount even with a read-only connection.

A WAL-mode index needs existing index.db-wal and index.db-shm files. Missing files are refused before opening, but this check cannot prevent their owner from removing them concurrently. Do not use SQLite’s immutable mode here: it would hide later commits in the live WAL.

Self::get stamps no hit and evicts or migrates nothing. A damaged or legacy entry is a miss. Mutators return an error, or do nothing for best-effort maintenance and memo writes. The thread’s durability policy, which Self::open sets, is left alone.

pub fn is_read_only(&self) -> bool

Whether this store was opened by Self::open_read_only.

pub fn file_hash_lookup(&self, path: &Path) -> FileHashLookup

Persistent-cache lookup for one file’s content hash — DB read only, no blake3. Lets the daemon’s HashFiles path release the store lock before the expensive file read (#281). See crate::file_hash::FileHashLookup.

pub fn file_hash_record(&self, fingerprint: &FileFingerprint, hash: &str)

Record a freshly-computed file content hash (the miss arm of Self::file_hash_lookup); best-effort.

pub fn record_verified_file_hash( &self, fingerprint: &FileFingerprint, hash: &str, )

Associate an already-known content hash with the exact fingerprint the caller observed it at, avoiding a redundant read when the file becomes a compiler input (kunobi-ninja/kache#540). Unlike Self::record_known_file_hash this never re-stats the path, so a file overwritten between observation and this call cannot inherit the old content’s hash.

pub fn record_verified_file_hashes(&self, restored: &[(FileFingerprint, &str)])

Self::record_verified_file_hash for every restored file of one hit, in one transaction with a short wait: the rows only save a later hash, so a busy index (a miss’s store transaction elsewhere) drops them instead of stalling the hit.

pub fn record_known_file_hash(&self, path: &Path, hash: &str)

Associate a stable file with its already-known content hash, avoiding a redundant read when it becomes a compiler input. Call this only after every store-side operation that may change the file’s fingerprint and while the compiler-owned output is stable.

pub fn file_hash_cache(&self) -> FileHashCache<'_>

Borrow the persistent memo without exposing the index connection.

pub fn contains(&self, cache_key: &str) -> bool

Check if a committed entry exists for this cache key.

pub fn stored_meta(&self, cache_key: &str) -> Option<EntryMeta>

An entry’s meta.json as written, with none of [Store::get]’s work: no blob verification, hit accounting or eviction. For a caller that just stored the entry and wants its file list.

pub fn get(&self, cache_key: &str) -> Result<Option<EntryMeta>>

Load metadata for a cached entry and record a hit.

pub fn try_lock(&self, cache_key: &str) -> Result<Option<KeyLock>>

Acquire a build lock for a cache key. Returns None if another process holds it.

pub fn claim_build(&self, cache_key: &str) -> Result<BuildClaim>

Claim a cache miss, re-checking the store after acquiring the key lock.

The re-check closes the window where a peer can commit and release its lock between this process’s cache lookup and lock acquisition.

pub fn try_gc_lock(&self) -> Result<Option<GcLock>>

Acquire the cross-process GC lock so concurrent GC drivers — a manual kache gc, the daemon’s periodic sweep, maybe_evict_after_upload, or a second daemon — don’t double-scan and contend. Returns None if another GC already holds it (the caller should skip).

pub fn try_durability_flush_lock(&self) -> Result<Option<StoreLock>>

The lock a durability flusher holds while it drains entries stored without an fsync. A wrapper that finds it held knows a flusher is already running for this store.

pub fn acquire_gc_lock(&self) -> Result<GcLock>

Block until the cross-process GC lock is held.

Durable upload-intent publication uses the same lock as every production GC driver: either GC finishes first and publication revalidates that the payload survived, or the intent becomes durable before GC snapshots its protected keys.

pub fn wait_for_committed(&self, cache_key: &str) -> Result<bool>

Wait for a cache key to become committed (another process is building it).

pub fn wait_for_committed_with_timeout( &self, cache_key: &str, timeout: Duration, ) -> Result<bool>

[Store::wait_for_committed] with a caller-chosen bound. false when the holder kept the key past timeout or released it without storing.

pub fn put_with_compile_time( &self, cache_key: &str, crate_name: &str, crate_types: &[String], features: &[String], target: &str, profile: &str, output_files: &[(PathBuf, String)], stdout: &str, stderr: &str, compile_time_ms: u64, ) -> Result<StorePutResult>

pub fn put_with_compile_time_independent( &self, cache_key: &str, crate_name: &str, crate_types: &[String], features: &[String], target: &str, profile: &str, output_files: &[(PathBuf, String)], stdout: &str, stderr: &str, compile_time_ms: u64, ) -> Result<StorePutResult>

Store outputs without ever sharing the compiler output inode with the read-only blob. Reflinks remain eligible because they provide CoW isolation; the fallback is a byte copy rather than a hardlink.

pub fn import_downloaded_entry(&self, cache_key: &str) -> Result<()>

Import a remotely downloaded entry into the database.

Downloaded entries arrive as tar archives extracted into the entry directory (old format: artifact files alongside meta.json). This method moves the artifact files into the content-addressed blob store and records them in the blobs table, leaving only meta.json in the entry directory.

pub fn import_restored_entry(&self, cache_key: &str) -> Result<()>

Import a restored entry into the local store.

This is the format-agnostic seam future remote layouts should call. A failed import must not leave an uncommitted meta.json behind: daemon download waiters use the extracted directory as a wake-up hint, and the residue could otherwise be mistaken for a published entry. Cleanup is serialized with Store publishers and preserves any committed generation that won the race.

pub fn import_verified_restored_entries( &self, entries: &[VerifiedRestoredEntry], ) -> Result<usize>

Import already stream-verified restored entries with one SQLite transaction for the whole batch.

Artifact content is not hashed again: callers may only construct VerifiedRestoredEntry after extraction verified each byte against meta.files[].hash. This method still performs a complete metadata and on-disk length preflight before moving blobs or opening the transaction.

pub fn blob_index_drift(&self) -> Result<BlobIndexDrift>

Verify that entry_blobs and blobs exactly match committed entry metadata. The write lock makes the filesystem/SQLite comparison stable.

pub fn blob_refcount_drift(&self) -> Result<BlobRefcountDrift>

Cheap check for drift between blobs and entry_blobs alone: plain reads, no meta.json and no write lock. See crate::BlobRefcountDrift for what it can and cannot see.

pub fn reconcile_blob_index(&self) -> Result<BlobIndexDrift>

Rebuild only the derived blob graph from committed entry metadata. Physical orphan reclamation deliberately happens after this transaction through Self::sweep_orphan_blobs, never while SQL can roll back.

pub fn reconcile_blob_index_dropping_unverifiable( &self, ) -> Result<(BlobIndexDrift, Vec<String>)>

Self::reconcile_blob_index that removes, in the same transaction, every committed entry whose metadata or blobs fail verification, and returns their keys. Such an entry can never be served: a lookup evicts it on sight. Kept, it makes the strict rebuild refuse the whole store. Blobs only those entries referenced lose their rows and are left to Self::sweep_orphan_blobs.

pub fn rebuild_index_from_store(&self) -> Result<RebuildStats>

Rebuild the entries and blobs rows by scanning the store’s per-entry meta.json files (kunobi-ninja/kache#415).

The index is derived state: the blobs plus each entry’s meta.json are the source of truth. So when the index is lost — quarantined after corruption, or deleted — the cache itself is still on disk and the rows can be reconstructed. Without this, recovery is needlessly lossy: a warm 100 GB cache silently becomes cold and every artifact is recompiled or re-downloaded even though the bytes never went anywhere.

Only registers an entry when every file it claims resolves to a blob that is present and the right size. A partially-present entry is skipped rather than registered, because a registered entry pointing at a missing blob is a false hit — strictly worse than a miss.

Idempotent, so it is safe to run on a populated index: entry rows are INSERT OR IGNOREd, and an entry already present contributes no blob refcounts. That matters because otherwise re-running would inflate every refcount and permanently leak blobs past their last referrer.

Deliberately does not re-hash blob contents. This is local, already content-addressed data, not the untrusted remote payload import_downloaded_entry validates; hashing a whole store would make recovery cost hours. doctor --verify --checksums remains the surface for content verification.

pub fn keys_for_crates( &self, crate_names: &[String], ) -> Result<Vec<CrateHistoryEntry>>

Look up cache keys for the given crate names (most recent per crate).

pub fn blob_path(&self, hash: &str) -> PathBuf

Resolve the filesystem path for a content-addressed blob. Layout: store/blobs/{first 2 hex chars}/{full hash}

pub fn matching_readonly_blob_inode( store_dir: &Path, output: &Path, ) -> Result<Option<PathBuf>>

Return the content-addressed blob whose inode a read-only output still shares, if any.

Older C/C++ restores could hardlink a build output to a read-only store blob. Passing that pathname to a compiler as root can truncate the blob and corrupt every cache entry that references it. This check is strictly read-only; callers fail closed instead of trying a racy path-based unlink.

pub fn sweep_stale_staging(&self, min_age: Duration) -> OrphanSweepStats

Reclaim crash-orphaned staging files older than min_age. A put killed between staging and publish leaves its snapshot here; unlike an orphaned blob it has no DB row to consult, so age is the only liveness signal — see STAGING_SWEEP_GRACE for why every caller wants the same one.

pub fn sweep_stale_key_locks( &self, min_age: Duration, cap: usize, ) -> Result<KeyLockSweepStats>

Unlink key lock files nobody needs: no entry row for the key, unused for min_age, and not held. At most cap per call; the rest wait for the next sweep. The caller holds gc.lock, so one sweeper runs at a time.

Each file is unlinked while this process holds its OS lock, and its age is read from the locked handle, so a claim that slipped in before the lock is seen. A contender that opened the file before the unlink finds the path changed once it gets the lock and reopens (StoreLock::acquire_current).

pub fn sweep_housekeeping(&self) -> HousekeepingStats

The per-key structures nothing else bounds: stale key lock files, unused input predictions and old file hash rows. The caller holds gc.lock. A pass that fails is logged and counts as zero; the next sweep tries again.

pub fn cache_dir(&self) -> &Path

Cache dir this store was opened with (blobs/, index.db, store/).

pub fn pending_durability(&self) -> Result<u64>

Entries stored without an fsync that no flush has reached yet.

pub fn flush_durability(&self, limit: usize) -> Result<usize>

Flush up to limit entries stored without an fsync: every blob and its shard directory, then the entry’s meta.json and directory, and mark them durable. An entry whose blob went missing meanwhile is evicted instead. Returns how many entries were flushed.

pub fn flush_entry_durability(&self, cache_key: &str) -> Result<bool>

Flush one entry stored without an fsync and mark it durable. Ok(false) when the entry was already durable or had to be evicted.

pub fn entry_dir(&self, cache_key: &str) -> PathBuf

pub fn total_size(&self) -> Result<u64>

Calculate the total size of the store.

pub fn physical_size(&self) -> Result<u64>

Registered blob content bytes: SUM(blobs.size), each deduplicated blob counted once. This — not Self::total_size’s logical per-entry sum — is what max_size bounds and what size pressure is measured against: the two diverge by exactly the dedup savings, which is largest in the cross-clone/worktree stores kache is aimed at (kunobi-ninja/kache#608). Not literally every byte under the cache dir: SQLite, meta.json files, and any blob whose best-effort unlink was deferred sit outside this sum.

pub fn blobs_at_least(&self, min_size: u64) -> Result<Vec<(String, u64)>>

Every indexed blob of at least min_size bytes, as (hash, size). One query, where listing the blob directories would stat every file.

pub fn entry_count(&self) -> Result<usize>

Get the number of entries in the store.

pub fn record_entry_unit(&self, cache_key: &str, unit: &str) -> Result<()>

Record which unit a freshly stored entry came from, by Cargo’s -C metadata hash. Every cache key folds that hash in, so a store with no row for (crate name, unit) holds no key the unit can produce; the crate-presence probe behind deferred discovery reads it.

pub fn remember_incremental_dir(&self, path: &Path) -> Result<()>

Remember an incremental compilation directory seen by the wrapper.

pub fn remember_target_root( &self, target: &Path, workspace_root: &Path, ) -> Result<()>

Remember a Cargo target root without putting absolute paths in cache entries or remote data. Updates are debounced to keep compiler-wrapper writes off the hot path.

pub fn remember_target_root_built_by( &self, target: &Path, workspace_root: &Path, rustc: Option<&str>, ) -> Result<()>

Self::remember_target_root, also recording the rustc -vV that built into it. None keeps whatever was recorded.

pub fn remember_discovered_target_root( &self, target: &Path, workspace_root: &Path, ) -> Result<bool>

Record a Cargo target found in a sibling Git worktree without claiming that a kache build used it. Never refresh an existing row’s last use.

pub fn tracked_target_roots( &self, stale_hours: u64, ) -> Result<Vec<TrackedTargetRoot>>

pub fn target_root_last_seen(&self, path: &Path) -> Result<Option<i64>>

When a build or Git discovery last recorded path, if it is tracked.

pub fn forget_target_root(&self, path: &Path) -> Result<()>

pub fn forget_target_roots_with_identity( &self, identity: PathIdentity, ) -> Result<()>

Forget the rows that record the directory identity names, under any spelling of its path, once a clean removed it. A row whose path still exists stays: that is another directory, such as one a build created since, that the filesystem gave the same inode.

pub fn forget_target_root_with_identity( &self, path: &Path, identity: PathIdentity, ) -> Result<()>

Self::forget_target_root, only while the row still records identity. A build may record a new directory at path after the caller read the row; that row stays.

pub fn clean_registered_incremental_dirs(&self) -> Result<usize>

Remove registered incremental directories and prune stale registry rows.

pub fn eviction_candidates(&self) -> Result<Vec<EntryFeatures>>

Materialize every entry’s eviction-relevant features in one pass.

Selection used to be three separate SELECTs embedded in three removal loops; it is now a pure function over these features (kunobi-ninja/kache#595). The size-pressure sweep already loaded every row, so this is the same I/O shape it always had.

pub fn eviction_candidates_for( &self, origin: SweepOrigin, ) -> Result<Vec<EntryFeatures>>

Self::eviction_candidates, with each entry’s import protection judged for a sweep started by origin.

pub fn evict(&self) -> Result<GcStats>

Weighted eviction: remove entries with lowest priority score until under the size limit. Prefers evicting old, rarely-accessed entries that actually free bytes.

Fires at max_size and evicts down to 90% of it — a real hysteresis band, not the single 90% line that used to serve as both trigger and target. The threshold lives here rather than at each call site so kache gc, the daemon’s periodic sweep, and the post-upload check all get the same band (see crate::eviction::over_eviction_trigger).

pub fn evict_for(&self, origin: SweepOrigin) -> Result<GcStats>

Self::evict for a sweep started by origin.

pub fn evict_for_disk_pressure( &self, stop: impl FnMut() -> bool, ) -> Result<GcStats>

Reclaim eligible entries even below the cache size budget. The caller holds gc.lock and stops on measured free space or a work deadline. Recent accesses, imports, upload intents and external retainers keep the same protection as other automatic sweeps.

pub fn evict_older_than(&self, hours: u64) -> Result<GcStats>

Evict entries older than the given duration.

pub fn evict_stale_key_schemas(&self, current_schema: u32) -> Result<GcStats>

Remove entries written by a different (or unknown legacy) cache-key recipe while retaining every entry from the running recipe.

This is deliberately explicit rather than part of ordinary GC: rows created before key-schema recording use 0, and an upgrade must not discard a still-reachable cache merely because its metadata predates this field. kache gc --stale-schema is the user’s opt-in boundary.

pub fn evict_duplicate_entries(&self) -> Result<GcStats>

Evict duplicate entries that share the same content_hash. Keeps the most recently accessed entry for each content_hash group (consistent with LRU eviction policy). Returns GcStats with eviction metrics.

Gated on the same size-pressure trigger as Self::evict. Reclaiming space is the only justification for spending a duplicate key’s hit history, so a comfortable store declines the sweep. Once triggered, the same physical-byte target bounds this pass; the following ordinary size sweep recomputes pressure if duplicate removal was insufficient.

pub fn evict_duplicate_entries_for( &self, origin: SweepOrigin, ) -> Result<GcStats>

Self::evict_duplicate_entries for a sweep started by origin.

pub fn sweep_orphan_blobs(&self, min_age: Duration) -> Result<OrphanSweepStats>

Reclaim orphaned blob files — content-addressed files on disk with no row in the blobs table. They accumulate when a crash interrupts a put/import between materialize (Phase 1) and the commit transaction (Phase 2), or when remove_entry runs against an entry whose meta.json is gone (so its blob hashes can’t be decremented). Nothing else reclaims them: evict*/remove_entry only touch blobs reachable from an entry, and total_size() doesn’t count them — so they leak invisibly to size-based eviction.

Only blobs whose file mtime is older than min_age are swept, so a blob a concurrent put is materializing (it renames the file into place just before inserting its row) is never reclaimed out from under it. Unlinks run while holding the SQLite write lock (BEGIN IMMEDIATE), upholding the store invariant that a blob is only ever removed under that lock — so even if this races a put adopting a long-lived orphan, that put’s Phase 2 re-materializes the blob before committing a reference to it.

pub fn backfill_content_hashes(&self) -> Result<usize>

Backfill content_hash for entries that don’t have one. Reads meta.json from each entry to get file hashes. Returns the number of entries updated.

pub fn backfill_compile_times(&self) -> Result<usize>

Backfill compile_time_ms for entries written before it was indexed (kunobi-ninja/kache#594), reading each entry’s meta.json — the same shape as Self::backfill_content_hashes, and run from the same GC sweep.

Only rows still at the 0 default are touched, so this converges: once an entry is backfilled it is never re-read. A genuinely zero-cost compile is indistinguishable from “not yet backfilled” here, which is harmless — it just gets re-read on the next sweep and stays 0.

Bounded to 10,000 entries per call. Measured on a real 52k-entry store, an unbounded pass is ~6 s of meta.json reads — and this runs inside the daemon’s GC sweep while the store mutex is held, so a first-GC-after-upgrade stall of that size is worth avoiding. Spreading it over successive sweeps costs nothing: eviction ranking treats a not-yet-backfilled entry exactly as it does today.

pub fn backfill_entry_blobs(&self) -> Result<usize>

Backfill entry_blobs rows for entries written before the table existed (kunobi-ninja/kache#608), reading each entry’s meta.json — the same shape and GC-sweep call site as Self::backfill_compile_times, and bounded the same way so a first-GC-after-upgrade never stalls on a 50k-entry store.

Converges: an entry gains rows once and is never re-read. Eviction ranks a not-yet-backfilled entry on its logical size, exactly as it did before the table existed. Entries whose meta.json is unreadable (or lists no files) can never gain rows and stay in the pre-#608 ranking regime; they are the same entries remove_entry already refuses to touch (#276). Selection is randomized so a batch of such entries cannot permanently starve the valid keys behind it.

pub fn shadow_counterfactual(&self) -> Result<ShadowCounterfactual>

What each policy’s evictions cost in later recompilation, over the sweeps both sides were recorded for (kunobi-ninja/kache#594).

The live side is every shadowed tombstone since the first recorded shadow victim. The shadow side is the tombstones it agreed with plus its own victims the live policy kept; a key it took earlier is not counted again when the live policy evicts it later. A shadow victim was demanded when its hit count rose after the sweep, on the entry or on its later tombstone, or when its tombstone saw a request. Rows of unknown cost are left out, as in Self::shadow_demand_split.

pub fn eviction_horizon_evidence( &self, as_of_unix_secs: i64, horizon_secs: u64, ) -> Result<EvictionHorizonEvidence>

Fixed-horizon observations from the latest live tombstone and first shadow-only decision per key. Does not change eviction or access state.

pub fn prune_tombstones(&self, keep_days: u64) -> Result<usize>

Drop tombstones older than keep_days, bounding the table.

Run from the GC sweep. A tombstone’s value is the demand signal in the window after eviction; past that it is only taking up space.

pub fn tombstone_stats(&self) -> Result<(usize, usize)>

(tracked, demanded) — how many evictions are being observed, and how many of those keys were later asked for again.

The ratio is the headline number for #594: a high rate means eviction is discarding entries the build still wants.

pub fn shadow_demand_split(&self) -> Result<ShadowDemandSplit>

Post-eviction demand split by the shadow policy’s verdict (kunobi-ninja/kache#594): of the entries the live policy evicted, how often was each cohort — “shadow agreed” vs “shadow would have kept” — later asked for again? A markedly higher demand rate on the would-have-kept cohort flags live-policy mistakes the shadow avoids. Both cohorts come from the same evicted population, so the comparison avoids the inventory-value circularity the issue warns about.

This is a live-victim diagnostic, not flip evidence on its own: the shadow’s own victims that the live policy KEPT are invisible here (their reuse shows up only as ordinary hits), rates are right-censored by tombstone age, and a flip decision needs the cost-weighted objective, not raw demand counts. Rows whose compile_time_ms is still 0 are recorded but excluded from the headline numbers: the density shadow ranks unknown-cost entries as worthless by construction, and freshness correlates with not-yet-backfilled, so counting them would bias the kept cohort with young, high-demand keys.

pub fn remove_entry(&self, cache_key: &str) -> Result<()>

Remove a single cache entry (files + DB record).

The entry row, its blob refcounts, and the unlink of any blob whose last reference is gone all happen inside one transaction. Because a blob file is only ever mutated while holding the SQLite write lock (here, and in put/import’s materialize step), the unlink can’t race a concurrent adopter: either we run first (the adopter re-materializes the file under the same lock) or it runs first (our decrement won’t reach zero).

pub fn clear(&self) -> Result<()>

Clear the entire store, including retained corrupt index copies. Tracked target directories stay registered: they live outside the store, and kache clean finds them through the registry.

Index rows drop first, in one transaction: once it commits no reader can begin a restore from a purged entry, and the filesystem wipe then runs against a store the index no longer references — a crash mid-wipe strands at worst orphan files for the sweep, never the pre-existing rows dangling over deleted blobs that the old wipe-then-delete order could leave.

Publishers don’t take gc.lock, so a put can still commit a fresh row while the wipe is deleting the files it just staged. The second row-deletion pass reduces that to the store’s tolerated shapes: a row committed before the pass is dropped (its files become sweepable orphans), and one committed after it at worst lands stranded — the refuse-removal / miss / re-put path that already recovers it.

pub fn list_entries(&self, sort_by: &str) -> Result<Vec<EntryInfo>>

List all entries for display.

pub fn migrate_to_blobs( &self, progress: impl Fn(usize, usize), ) -> Result<MigrationStats>

Bulk-migrate all legacy entries’ artifacts into the blob store.

pub fn blob_stats(&self) -> Result<BlobStats>

Return content-dedup statistics: unique blobs, physical vs logical size.