Crate limnifs_write 

LimniFS writer pipeline — directory tree to .lim image.

The writer takes a real directory tree and produces a valid .lim manifest artifact with inlined metadata. Files at or below the inline threshold (4 KiB) are stored as inline data in their inodes; larger files are stored as drops packed into a single slab.

Usage

use std::path::Path;
use limnifs_write::write_directory;
let artifact = write_directory(Path::new("/path/to/dir"))?;
std::fs::write("output.lim", &artifact.bytes)?;

Re-exports

pub use config::profile;

pub use config::CategorizerConfig;

pub use config::ChunkingConfig;

pub use config::CodecRegistry;

pub use config::CodecTunables;

pub use config::Defaults;

pub use config::DictionaryConfig;

pub use config::EncryptionConfig;

pub use config::TournamentConfig;

pub use config::WriteConfig;

Modules

chunker

Content-defined chunking via FastCDC (Xia et al., 2016).

classifier

Drop classifier (seine): entropy + magic-byte heuristics.

compaction

Slab compaction — removes unreferenced drops from a slab without re-reading file contents. Preserves codecs, drop identities, and metadata; only the slab and manifest’s slab index / history change.

config

User-facing write configuration.

delta_builder

Delta builder — computes tree operations between two images.

dictionary

ZSTD dictionary training — writer-side API.

file_categorizer

File-level categorizer framework.

flatten

Metadata-only flattener — merge N manifests into a single composite manifest with zero drop-store I/O.

rw

Read-write image API — LimniFS’s key differentiator.

turnover

Turnover — tier-3 full re-encode defrag.

Structs

MetadataSidecar

Externalized metadata sidecar, present when the metadata blob exceeds METADATA_EXTERNALIZE_THRESHOLD. Callers must write bytes to locator next to the manifest file.

SlabArtifact

One slab produced by the writer. The slab ordinal in id matches the slab’s position in WriteArtifact::slabs.

WriteArtifact

Result of writing a directory tree.

Enums

WriteError

Error during writing.

Constants

INLINE_THRESHOLD

Inline-data threshold: files at or below this size get inline data in their inode. Larger files are stored as drops in a slab.

MAX_SLAB_TOTAL_BYTES

Maximum total length of a single slab file (header + content). Matches the reader’s DEFAULT_SLAB_MAX_BYTES (spec §3.1) minus a safety margin so a slab that is full but not yet flushed cannot overrun the reader ceiling on the next drop.

METADATA_EXTERNALIZE_THRESHOLD

Default threshold at which the writer externalises the metadata blob to a sidecar file instead of inlining it in the manifest. Derived from the reader’s inline ceiling (limnifs_core::metadata_reference::DEFAULT_INLINE_METADATA_MAX_BYTES, 1 MiB per spec §5.3) minus headroom, so the two constants cannot silently drift apart. Override per image via WriteConfig::defaults::metadata_externalize_threshold (issue #187).

METADATA_LARGE_BLOB_QUALITY

Brotli quality for large metadata blobs. q2 is much faster than q5 on multi-MiB inputs; ratio on highly compressible inode data is within 5–10% of q5 (often identical) because metadata is dominated by long runs of repeated patterns.

METADATA_LARGE_BLOB_THRESHOLD

Metadata-blob size above which the writer steps Brotli quality down to METADATA_LARGE_BLOB_QUALITY. Below this, q5’s cost is negligible; above it, q5 starts to dominate create time on big inode trees (e.g. the 50 K-file tiny-files dataset).

METADATA_SMALL_BLOB_QUALITY

Brotli quality for small metadata blobs (≤ METADATA_LARGE_BLOB_THRESHOLD). Best ratio; cost is in the noise on small inputs.

MMAP_READ_THRESHOLD

Above this size, mmap the input file instead of std::fs::read-ing it into a Vec<u8>. Keeps peak RSS bounded when packing huge files (multi-GiB source trees, ML models). Crossover is around 1 MiB on most filesystems — below that the syscall + VMA setup costs more than the read.

WHOLE_FILE_MAX_SIZE

Maximum file size for the whole-file categorizer path. Files above this threshold use FastCDC chunking even when a categorizer claims them, enabling rayon parallelism across chunks. The categorizer’s codec is still used per-chunk when possible.

Functions

write_directory

Walk a directory tree and produce a valid .lim manifest artifact with inlined metadata. Files at or below INLINE_THRESHOLD bytes are stored inline; larger files are packed into a single slab as content-addressed drops.

write_directory_with_config

Create an image with a custom WriteConfig (e.g. from a profile).

write_layer

Pack a directory tree as a layer on top of a base image.

write_stream

Pack a single named stream into a .lim image.