Module bytecode_cache

Expand description

Content-addressed on-disk cache for compiled .harn pipelines.

Cold-start harn run re-parses, type-checks, and compiles the entry pipeline before the VM gets a single instruction to execute. For short Harn subcommands that wrap a few llm_calls in a small pipeline, that compile cost dominates wall-clock time.

This module persists Chunk bytecode under $HARN_CACHE_DIR/<source-hash>.harnbc (XDG-aware). The cache key is derived from the source plus its compilation context. Entry chunks include the content of every transitively-imported user file because they compile the complete program. Module artifacts compile exactly one file and retain unresolved import specs, so their context includes compiler and embedded stdlib identity but deliberately excludes user dependencies. Any change to an artifact’s actual compilation inputs flips the key and recompiles it.

File layout — little-endian throughout:

magic        : [u8; 8]   = "HARNBC\0\0"
schema_ver   : u32       = SCHEMA_VERSION
version_len  : u32
harn_version : [u8; version_len]
compiler_tag : u8        bitmask of active CompilerOptions
kind         : u8        1 = entry chunk, 2 = module artifact
source_hash  : [u8; 32]
context_hash : [u8; 32]
payload      : postcard-serialized payload for `kind`

The header lets a stale binary detect a future-version artifact without crashing: a magic mismatch, schema mismatch, or version mismatch is returned as Ok(None) so the caller transparently recompiles. Real I/O errors propagate.

Concurrency: writes go through crate::atomic_io (write-tmp, fsync, rename, fsync parent dir), and parallel invocations on a cache miss race safely — the last writer wins, but every reader observes a consistent file because the rename is atomic on every supported filesystem.

Structs§

CacheKey: Cache key components for a single pipeline source. Equality of all fields is necessary and sufficient for cache reuse.
LookupOutcome: Result of a cache lookup. Carries the precomputed key so the caller can write it back on a miss without rehashing.
ModuleLookupOutcome: Result of a load_module lookup. Carries the precomputed key so the caller can write it back on a miss without rehashing.

Constants§

CACHE_DIR_ENV: Environment override for the cache directory. When set, takes precedence over the XDG and home-directory fallbacks.
CACHE_ENABLED_ENV: Environment override that turns the cache off entirely. Setting this to 0, false, no, or off skips both reads and writes; useful when debugging compiler changes.
CACHE_EXTENSION: Conventional extension for entry-chunk cache files.
CODEGEN_FINGERPRINT: Build-time fingerprint of the compiler front-end — the lexer, parser, IR, and code generator — computed in build.rs from those crates’ source and baked in via cargo:rustc-env. Folded into the cache key so a compiler change that alters emitted bytecode for unchanged source invalidates stale entries automatically, within a single version, with no manual cache wipe. HARN_VERSION only busts the cache across release bumps; this closes the same gap for the within-version compiler edits that masked #2610. See #2621.
HARN_VERSION: Compile-time Harn release. Cache files written by a different release are rejected on load.
MAGIC: Header magic for all bytecode-cache artifact families.
MODULE_CACHE_EXTENSION: Conventional extension for module-artifact cache files. Distinct from CACHE_EXTENSION so the same .harn source can have both shipped adjacent if needed (e.g. when a file is both an executable entry and imported by other files).
SCHEMA_VERSION: On-disk format version. Bump when CachedChunk or the header layout changes in a backwards-incompatible way. v5: ModuleArtifact gained public_type_names (pub type exports). v6: payload encoding replaced with postcard. v7: exported type schemas moved from eager JSON strings to an initializer chunk that resolves imported aliases in the module environment.

Functions§

adjacent_cache_path: Path to the adjacent precompiled entry-chunk artifact for source_path. foo.harn → foo.harnbc.
adjacent_module_cache_path: Path to the adjacent precompiled module-artifact for source_path. foo.harn → foo.harnmod.
cache_dir: Returns the directory the shared cache lives in. Honors $HARN_CACHE_DIR, then $XDG_CACHE_HOME/harn/bytecode, then $HOME/.cache/harn/bytecode. The directory is not created here — store creates it lazily on write so read-only environments don’t pay an mkdir cost.
cache_enabled: True when the cache is enabled by the current environment.
load: Try to load a cached chunk for source_path whose contents are source. Returns the key alongside the (optional) chunk so callers avoid recomputing the key on miss.
load_module: Look up the ModuleArtifact for source_path (whose contents are source). Mirrors load but for the .harnmod family.
packs_cache_dir: Root for .harnpack archives unpacked by harn run <bundle.harnpack>. Each verified bundle is replayed into <root>/<sanitized-bundle-hash>/ so re-runs reuse the unpacked tree. Honors $HARN_CACHE_DIR/packs when set, otherwise XDG / $HOME/.cache/harn/packs.
serialize_chunk_artifact: Serialize an entry-chunk artifact (header + payload) to bytes. The resulting buffer is byte-identical to the file store_at would have written for the same (key, chunk). Use this when packaging artifacts into a container (e.g. harn pack) without going through the filesystem.
serialize_module_artifact: Serialize a module artifact (header + payload) to bytes. Companion to serialize_chunk_artifact for the .harnmod family.
store: Persist chunk to the shared cache directory under key. Atomic: a temp file is written then renamed into place. Concurrent invocations on the same key race safely.
store_at: Write a precompiled entry-chunk artifact to an explicit path, for use by the harn precompile subcommand. The header still records the key, so adjacent artifacts shipped with source are validated like any other cache hit.
store_module: Persist artifact to the shared cache under key. Atomic; concurrent invocations race safely.
store_module_at: Write a module artifact to an explicit path.