# timberfs
Experiment: a Linux filesystem purpose-built for log files.
Log files have a very particular access pattern that general-purpose
filesystems don't exploit:
- **append-only writes** — nothing ever rewrites the middle of a log
- **highly compressible content** — typically 10–20x with zstd
- **time-correlated reads** — "what happened between 13:42 and 13:43?" is
*the* question, but answering it on a plain file means scanning gigabytes
`timberfs` is a FUSE filesystem that presents ordinary-looking log files while
storing them chunked + zstd-compressed, with a per-chunk **write-time index**
so a time-range query is a binary search + a few frame decompressions —
independent of file size.
## Why FUSE (and not overlayfs / a kernel module)
- **overlayfs** layers *namespaces* (upper/lower directories, as used by
container images). It has no hook for transforming *content*, so it can't
compress on write or maintain an index. Wrong tool.
- **A native kernel filesystem in Rust** is where Rust-for-Linux is heading,
but the filesystem bindings are still experimental. Not a good vehicle for
iterating on a design.
- **FUSE** gives us the full VFS interface in userspace: loggers append
through the mount unmodified, `tail -f`/`grep`/`less` all just work, and
the implementation is ordinary safe Rust (`fuser` crate, no libfuse
dependency — it only needs the `fusermount3` binary at runtime).
The design cleanly splits into a *store* (file format + chunking, no FUSE
types) and a thin FUSE layer, so the store could later be re-hosted in a
kernel module, a `LD_PRELOAD` shim, or a log-shipping daemon without change.
## On-disk format
Each logical file `<name>` is backed by two files in the backing directory:
```
<name>.trunk concatenated zstd frames, one per chunk, no wrapper bytes
<name>.rings 8-byte magic "RING0001", then 48-byte records (all u64 LE):
uncomp_start | uncomp_len | comp_start | comp_len
| first_write_ms | last_write_ms
```
(The names take the timber metaphor seriously: the data is the trunk, and
the index is its growth rings — which really are a write-time index;
dendrochronology dates events by rings exactly the way `timberfs query`
dates bytes by chunks.)
Because the `.trunk` is a plain zstd frame concatenation, **stock tools can
always recover the data**: `zstd -dc app.log.trunk` prints the whole
uncompressed log, no timberfs required. The index is pure acceleration.
Records are appended in write order, so they are sorted both by uncompressed
offset **and** by wall-clock time — byte reads and time queries are each one
`partition_point` binary search.
Crash safety: chunks are written data-first, index-second; on open, index
records pointing past the end of the data are dropped and orphaned data
bytes are overwritten. `fsync()` through the mount flushes the buffer as a
chunk and syncs both backing files, so fsync = durable. Unsynced buffered
data is lost on a crash, bounded by `--flush-age`.
## Semantics
| append (write @ EOF) | buffered, compressed into a chunk on size/age/close/fsync |
| write elsewhere | `EPERM` — the filesystem is append-only |
| read anywhere | chunk located by binary search, decompressed, served |
| truncate to 0 | allowed: starts the file over (copytruncate-style rotation) |
| truncate elsewhere | `EPERM` |
| rename / unlink | supported (mv-based log rotation works) |
| `ls -l` size | logical (uncompressed) size |
| `du` blocks | compressed size — `du -h` shows the real disk footprint |
| subdirectories | not yet — flat namespace in v0 |
Time-range queries are **chunk-granular** — by design, not as a
placeholder: every chunk whose write-time window overlaps the requested
range is returned in full. Chunk windows are bounded by `--flush-age`
(default 5 s) for slow writers and by `--chunk-size` (default 256 KiB) for
fast ones, so that's the worst-case slop at the edges of the window.
The intended workflow is: `timberfs query` does the coarse seek into a huge
file (cheap, no parsing, immune to multiline entries and timestamp-less
lines), then `timberfs grep` (entry-aware) or ordinary `grep`/`awk` on the
small extract trims exactly using the timestamps the log lines carry
anyway. The slop is a feature there:
buffered loggers write lines slightly after the timestamp they print, so a
byte-exact write-time cut could miss edge lines that grep-on-content
catches.
## Usage
```sh
# mount: logical view on ./logs, compressed store in ./logs-backing
timberfs mount ./logs-backing ./logs &
# any process just appends normally
myapp >> logs/app.log
echo "hello" >> logs/app.log
tail -f logs/app.log
grep ERROR logs/app.log
# the killer feature: extract by wall-clock write time, O(log n)
timberfs query logs-backing/app.log --from 13:42 --to 13:43
timberfs query logs-backing/app.log --from "2026-07-09 13:42:00" --to "2026-07-09 13:43:00"
# entry-aware grep: matches whole log ENTRIES (a timestamped line plus
# its continuations — stack traces stay attached); pipe for AND
timberfs grep ERROR logs-backing/app.log --from 13:42 --to 13:43
# inspect the chunk index (offsets, compression ratio, time windows)
timberfs index logs-backing/app.log
# quick metadata via xattrs on the mounted file
getfattr -d -m 'user.timberfs.' logs/app.log
# escape hatch: recover everything with stock tools, no timberfs needed
zstd -dc logs-backing/app.log.trunk
# unmount
fusermount3 -u ./logs
```
### Piping without FUSE
The mount is optional: `timberfs append` writes the same store directly
from a pipe — the daemontools/runit/s6 log-processor pattern (`multilog`,
`svlogd`, `s6-log`), so it drops into supervision trees and containers
where FUSE is unwelcome (no `/dev/fuse`, no root, no mount):
```sh
```
Each log has exactly one writer (a per-file lock), appenders for different
files share a directory freely, and a directory is either mounted *or*
appended to — never both (the mount daemon owns in-memory state for the
whole directory). End of input, `SIGTERM` or `SIGINT` flush and sync
everything before exit.
**Importing existing logs** works the same way, but with a twist that
matters: chunk time windows come from timestamps *parsed out of the log
lines* (auto-detected RFC3339/ISO, Apache/CLF, or leading epochs;
`--timestamp-regex`/`--timestamp-format` for anything else), because the
write time of historical data says nothing. Lines without a timestamp —
stack traces, continuations — inherit the previous line's, and mildly
out-of-order lines just widen chunk windows (queries select by interval
overlap, so nothing is lost):
```sh
timberfs import /var/log/old-app.log logs-backing/app.log
timberfs query logs-backing/app.log --from "2026-06-03 14:00" --to "2026-06-03 15:00"
# a whole rotated set, in any order — files are stitched chronologically
# by their own first timestamps (rotation numbering and glob order lie)
timberfs import /var/log/old-app.log.* /var/log/old-app.log logs-backing/app.log
# a timberfs source (say, a rotation segment shipped from another box)
# is detected automatically and merged VERBATIM — no decompression, no
# parsing, index included; re-shipping the same segment is a no-op
timberfs import /shipped/app-2026-07-09.log central-backing/hostA-app.log
```
And `timberfs export` is the read-side twin: carve any time window (or a
whole log) out of an archive as a fresh timberfs log — or as a
single-file **`.timber` bundle** for shipping (a plain uncompressed tar,
`.rings` member first, so `tar xf` + `zstd -dc` always recovers it and a
hand-tarred pair is a valid bundle):
```sh
timberfs export backing/archive.log incident.timber --from 13:40 --to 14:10
```
Note the middle line: bundles are first-class *read-only* logs — `query`,
`index` and `export` operate on a `.timber` file directly (tar keeps its
members contiguous and uncompressed, so the trunk member is just a trunk
at an offset). A directory of `.timber` case files is a queryable cold
archive; unpacking or importing is only ever needed to append.
Together these close the shipping loop: `rotate` cuts history out of live
logs, `export` carves arbitrary windows out of anything, `.timber` bundles
travel as single files, and `import` merges them anywhere, idempotently —
every step a verbatim copy of compressed chunks. The shipping format *is*
the storage format.
Re-importing is idempotent: the target is its own checkpoint. Already
imported bytes are verified against the source (all chunks, or
first/middle/last with `--quick`), then only the growth is appended —
an unchanged source is a no-op, a rotated/rewritten one is refused before
anything is written. So a periodic `timberfs import` of a growing file
is a safe, cheap catch-up (full verification of a multi-GB target runs
in well under a second).
The appender is also where **retention** lives, because it already owns
the file: `--retain 30d` continuously drops data older than 30 days, and
`--retain-size 200G` keeps the compressed on-disk size under a hard
budget, oldest first — combine them for "keep the last 30 days, but never
more than 200G":
```sh
No dated-file rotation needed: the log is simply a single file that always
contains the recent past, and `timberfs query` finds things in it by time.
(Head-dropping currently compacts by rewriting the retained data, so
enforcement is batched — expired data goes once it's ~10% of the file, size
overruns trim to 95% of budget — and compaction briefly needs free space
proportional to what's kept. Hole-punching is the planned fix for very
large stores.)
## Rotation & retention
`timberfs rotate` does **time-based** rotation: everything written before the
cutoff moves out of the live log into another one (or is dropped), while
newer data stays put — a cut a normal filesystem can't do without rewriting
the whole file.
```sh
timberfs rotate backing/app.log app-2026-07-08.log --cutoff "2026-07-09T00:00"
timberfs rotate backing/app.log --delete --cutoff "2026-06-01T00:00" # retention
timberfs rotate backing/app.log archive.log --cutoff 12:00 --dry-run # preview
```
Why it's cheap: chunks are immutable zstd frames, so rotation relocates
**compressed bytes verbatim** — no decompression, no recompression — and
rebases the index records. Rotating gigabytes of logs costs I/O proportional
to the compressed size. The destination (same backing directory) is created
or appended to; appends are refused if they would break the index's time
ordering. Like queries, the cutoff is chunk-granular: a chunk straddling it
stays in the live file.
It works against a live mount: the daemon holds an `flock` on
`<backing>/.timberfs.lock` recording its mountpoint, and `timberfs rotate`
auto-detects it — offline it rewrites the backing files directly (holding
the same lock), mounted it routes the request through the daemon as a
`setxattr` control call (`user.timberfs.rotate`), which rotates atomically
under the daemon's state lock and then invalidates the kernel's attribute
cache so writers holding the file open with `O_APPEND` keep working across
the shrink (their next write re-bases to the new EOF).
`timberfs query`/`timberfs index` read the backing files directly and are safe to
run against a live mount (chunks are immutable, the index is append-only).
Note they only see flushed chunks — the still-buffered tail (≤ flush-age
old) is visible through the mount but not yet in the backing files.
## Custom indexes: the .grain token index
The write-time index generalizes: `.rings` is just a per-chunk summary
(byte ranges + a searchable time window), and queries never touch the
trunk except for the chunks the summary selects. The first content index
is implemented: **`.grain`**, one Bloom filter per chunk over every token
in it (~10 bits per distinct token, k=7, ~1% false positives — measured
0.86% on a 2.7 GB production log). Build it with `timberfs reindex`, use
it with `query --has`:
```sh
timberfs reindex logs-backing/app.log # 2.7 GB indexed in ~6 s
timberfs query logs-backing/app.log --has F454567068093ZHGZCL # no time bound!
timberfs query logs-backing/app.log --from 13:00 --to 14:00 --has ERROR \
| timberfs grep 'tenantId=FOO'
```
Tokens are ASCII-alphanumeric runs of 3–64 characters, exact case,
config-free: rare tokens (request keys, message ids) skip nearly every
chunk, ubiquitous ones skip nothing and cost only the test. `--has` is a
**chunk-level pre-filter with whole-token matching** — an argument with
separators (`req-8f3a`) must match all its tokens in the same chunk, AND
across repeated `--has` flags is also chunk-level, and substrings of
tokens do not match; exact, entry-level filtering stays downstream in
`timberfs grep`. A false positive costs one needless chunk
decompression. The design contract that made this a sidecar:
**Custom indexes are sidecars**: one file per index next to the `.trunk`/`.rings` pair (the
metaphor extends: `.rings` is time, content indexes are *grain*), with a
self-describing header (index type + extractor description) and one
append-only entry per chunk. Three rules:
1. **Derived and rebuildable.** A sidecar can always be regenerated by
streaming the trunk (`timberfs reindex`), so indexes can be added to
existing logs, reconfigured, or deleted at zero risk. The trunk and
rings remain the only durable truth.
2. **Missing means scan.** A chunk without an index entry is "no
information — scan it". Partial or lagging indexes degrade to
conservative scans, never wrong answers; this is also the crash story.
3. **Rings rewrites delete sidecars.** Any operation that rewrites the
`.rings` (rotation, retention head-drop) deletes the file's custom
indexes; `reindex` recreates them. No coordination logic, no corruption
class. (Prefix-trimming sidecars in the same pass is a later
optimization, since head-drops remove exactly a chunk prefix.)
Consequences worth knowing: chunk size is an index-selectivity knob
(smaller chunks → sharper lookups, more overhead), the grain lags a live
appender until the next `reindex` (lagging entries just mean scanning
those chunks), and `.timber` bundles carry no grain yet. A
chunk-sequence-number field in the rings header was considered to let
sidecars survive head-drops without deletion, and rejected: rule 3 makes
it unnecessary, and the on-disk format stays `RING0001`. (Logged-timestamp
zone maps, the other planned index family, became largely moot: import
already writes logged time into the rings.)
## Install
Debian/Ubuntu, from the apt repository (rebuilt by CI from the GitHub
releases on every release, GPG-signed, `apt upgrade` works):
```sh
sudo curl -fsSL https://torstei.github.io/timberfs/key.gpg \
-o /usr/share/keyrings/timberfs.gpg
sudo tee /etc/apt/sources.list.d/timberfs.sources >/dev/null <<'EOF'
Types: deb
URIs: https://torstei.github.io/timberfs
Suites: stable
Components: main
Signed-By: /usr/share/keyrings/timberfs.gpg
EOF
sudo apt update && sudo apt install timberfs
```
Or grab a single `.deb` from the latest GitHub release (built, VM-tested
and provenance-attested by CI — verify with
`gh attestation verify timberfs_amd64.deb --repo torstei/timberfs`):
```sh
curl -LO https://github.com/torstei/timberfs/releases/latest/download/timberfs_amd64.deb
sudo apt install ./timberfs_amd64.deb
```
Or from crates.io with a Rust toolchain: `cargo install timberfs`.
## Build
Needs the Rust toolchain and a C compiler (for the vendored zstd), plus
fuse3 at runtime:
```sh
sudo apt install rustup build-essential fuse3 # or rustup.rs installer
rustup default stable
cargo build --release # target/release/timberfs
```
### Debian package
```sh
cargo install cargo-deb
cargo deb # target/debian/timberfs_*.deb
sudo dpkg -i target/debian/timberfs_*.deb
```
The package installs `/usr/bin/timberfs` plus a systemd template unit: drop
a config in `/etc/timberfs/<instance>.conf` (see
`/usr/share/doc/timberfs/examples/timberfs.conf.example`) and run
`systemctl enable --now timberfs@<instance>` to mount at boot. Stopping the
unit unmounts first, so the daemon flushes everything and exits cleanly.
## Ideas / future work
- **zstd seekable format / dictionaries**: adopt the official seekable-zstd
framing for ecosystem compat; train a dictionary per file for much better
small-chunk ratios; long-range mode for cold recompression.
- **Cold-chunk recompression**: rewrite old chunks at zstd -19 in the
background; the index makes this a local, safe operation.
- **Scheduled rotation**: a `timberfs rotated`-style timer (or systemd timer
recipe) driving `rotate --cutoff`/`--delete` policies per file.
- **Appender growth toward s6-log**: `SIGHUP`-triggered and scheduled
rotation into dated files (for shipping archives off-box), optional line
timestamping, `--tee` passthrough, and a `--follow` reader.
- **Hole-punching retention**: drop the head via `FALLOC_FL_PUNCH_HOLE`
instead of compact-rewrite, making `--retain-size` cheap on huge stores.
- **Expose the index in-band**: a virtual `.idx` twin file or ioctl so tools
can query through the mount without knowing the backing dir.
- **tail(1) fast-path**: negative-offset "time seek" via `llseek` hooks.
- **Subdirectories, multi-writer O_APPEND atomicity, runtime rescan of the
backing dir, real statfs passthrough.**
- **Kernel port**: the store layer is FUSE-free by design; revisit
Rust-for-Linux filesystem bindings when they stabilize.
## License
Licensed under either of
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT license ([LICENSE-MIT](LICENSE-MIT))
at your option.
Unless you explicitly state otherwise, any contribution intentionally
submitted for inclusion in the work by you, as defined in the Apache-2.0
license, shall be dual licensed as above, without any additional terms or
conditions.