.TH TIMBERFS 1 "2026-07-10" "timberfs 0.1.0" "User Commands"
.SH NAME
timberfs \- append-only, transparently compressed, write-time-indexed filesystem for log files
.SH SYNOPSIS
.B timberfs mount
.RI [ options ]
.I BACKING
.I MOUNTPOINT
.br
.B timberfs create
.RB [ \-\-index ]
.RB [ \-\-set
.IR KEY=VALUE ]...
.I DEST
.br
.B timberfs append
.RI [ options ]
.I FILE
.br
.B timberfs import
.RI [ options ]
.B \-\-into
.I DEST
.IR SOURCE .\|.\|.
.br
.B timberfs export
.RB [ \-\-from
.IR TIME ]
.RB [ \-\-to
.IR TIME ]
.I SOURCE
.I DEST
.br
.B timberfs query
.RB [ \-\-from
.IR TIME ]
.RB [ \-\-to
.IR TIME ]
.IR FILE .\|.\|.
.br
.B timberfs grep
.RI [ options ]
.I PATTERN
.RI [ FILE .\|.\|.]
.br
.B timberfs index
.I FILE
.br
.B timberfs reindex
.I FILE
.br
.B timberfs rotate
.B \-\-cutoff
.I TIME
.I SOURCE
.RI [ DEST ]
.RB [ \-\-delete ]
.RB [ \-\-dry\-run ]
.SH DESCRIPTION
.B timberfs
is a FUSE filesystem purpose-built for log files. Files in the mount look
and behave like ordinary log files \(em loggers append,
.BR tail (1)
and
.BR grep (1)
work unmodified \(em but the data is stored in a backing directory as
chunks compressed with zstd, together with a per-chunk
.B write-time index
that records when each chunk was written. Asking \(lqwhat was written
between 13:42 and 13:43?\(rq is a binary search plus a few frame
decompressions, independent of file size.
.PP
Files are append-only: writes anywhere but the end of the file fail with
.BR EPERM .
Truncating a file to zero is allowed and starts it over
(copytruncate-style rotation); rename and unlink work normally.
.B ls \-l
shows the logical (uncompressed) size while
.B du
shows the real, compressed disk usage.
.SH SUBCOMMANDS
.SS mount
.B timberfs mount
.I BACKING MOUNTPOINT
serves the logical view of backing directory
.I BACKING
on
.IR MOUNTPOINT ,
in the foreground. Unmount with
.B fusermount3 \-u
.IR MOUNTPOINT .
.TP
.BI \-\-chunk\-size " BYTES"
Uncompressed buffer size that triggers a chunk flush (default 262144).
.TP
.BI \-\-level " N"
zstd compression level (default 3).
.TP
.BI \-\-flush\-age " SECONDS"
Maximum time appended data may sit unflushed (default 5). Bounds both the
time granularity of the index and the data lost in a crash without
.BR fsync (2).
.TP
.B \-\-allow\-other
Let other users access the mount. Unprivileged users need
.B user_allow_other
in
.IR /etc/fuse.conf ;
root (e.g. the systemd unit) does not.
.SS create
.B timberfs create
.I DEST
makes an empty log with its properties declared up front in a
.B .bark
manifest \(em one flat, optional, human-editable JSON object beside the
pair.
.B \-\-index
declares the token index, database-style: once declared, EVERY import
maintains the
.B .grain
automatically (extended incrementally for new chunks, rebuilt if it went
missing \(em say, dropped by rotation or retention), so there is no
per-import flag to forget.
.B \-\-set
.I KEY=VALUE
records free-form provenance
.RB ( host ,
.BR path ,
.BR format ,
\|.\|.\|.). Every manifest is minted with a durable identity on first write:
.B id
(a random UUID, constant across renames, moves and hosts \(em the
identity of the store, where paths are merely its current address) and
.B created
(RFC3339). Artifacts made by
.B export
and by rotation into a new segment are NEW stores: they get a fresh
identity, a
.B derived_from
pointer to the source's id (lineage chains compose across re-carves and
shipping), a
.B derived_op
of
.BR export / rotate
(copy vs move \(em whether the source still holds the data), and inherit
the data-provenance fields; settings
.RB ( index )
do not inherit. Rotation mints its source's identity when missing (it
holds the writer locks); export never writes its source. Content facts
\(em actual spans, sizes \(em are deliberately NOT recorded in bark: the
artifact's own rings state them authoritatively. The REQUESTED window is
different \(em an operation fact, like
.B derived_op
\(em and a windowed export records it as
.BR window_from / window_to
(RFC3339 UTC; only the bounds given; never inherited). Content can never
state coverage \(em an artifact whose last line is 17:00 doesn't say
whether 17:00\(en24:00 was covered-but-silent or not covered \(em and for
an EMPTY artifact the declared window is its entire meaning. Unlike the
derived
.BR .grain ,
bark survives head-drops, travels on rename, and ships inside
.B .timber
bundles.
.SS append
.B timberfs append
.I FILE
reads standard input and appends it to a log in the backing store directly,
with no FUSE mount involved \(em the
.BR svlogd (8)/s6-log
pattern:
.PP
.nf
.RS
myapp 2>&1 | timberfs append logs\-backing/app.log
.RE
.fi
.PP
Each log has exactly one writer (an exclusive per-file lock), but
appenders for different files happily share a backing directory; a
directory served by a
.B timberfs mount
is unavailable to appenders and vice versa. Appending to an existing log
continues it. Data is flushed into chunks by the same size and age rules
as the mount; end of input,
.B SIGTERM
or
.B SIGINT
flush and sync everything before exit. Accepts the same
.BR \-\-chunk\-size ,
.B \-\-level
and
.B \-\-flush\-age
options as
.BR mount ,
plus:
.TP
.BI \-\-retain " DURATION"
Continuously drop data older than
.I DURATION
(e.g.
.BR 30d ,
.BR 12h ,
.BR 90m ).
.TP
.BI \-\-retain\-size " SIZE"
Keep the on-disk (compressed) size of the log at or under
.I SIZE
(e.g.
.BR 200G ,
.BR 512M ;
powers of 1024), dropping the oldest data first. Combines with
.BR \-\-retain :
whichever limit bites first wins.
.PP
Retention is checked every second, at startup and once more at shutdown.
Dropping the head currently compacts the log by rewriting the remaining
data, so enforcement is batched: age-expired data is dropped once it makes
up about a tenth of the file, and a size overrun is trimmed down to 95% of
the budget. Compaction briefly needs free disk space proportional to the
retained data.
.SS import
.B timberfs import \-\-into
.I DEST
.IR SOURCE .\|.\|.
converts existing plain log files into a timberfs log. The destination is
a named flag on purpose: a shell glob among the sources can never
swallow it. Several sources \(em a rotated set \(em may be given
in any order: each file's first parsed timestamp determines the stitch
order (rotation numbering and glob order lie), and the plan is printed
before anything is written. Two sources starting at the same timestamp
are refused as likely duplicates.
.PP
Each plain source is placed by its
.B first timestamp
relative to what the destination already holds. Starting
.I after
the destination's last write, it simply appends (the daily bulk-load).
Starting
.I inside
the destination's window \(em day files cut with slack, an import re-run
\(em the overlap is deduplicated
.BR "line by line" :
lines the destination already has are skipped, genuinely new lines in
the covered window are imported (with a warning, since that usually
means the captures disagreed), and re-importing an already-covered file
is a clean no-op. Counts are reported. Starting
.I before
everything in the destination is refused \(em import in chronological
order or to a new target. A source starting exactly
.B where the destination starts
is treated as the same file, regrown: the already-imported prefix is
byte-verified and only the growth is appended (truncated or rewritten
files are refused before anything is written; see
.BR \-\-quick ).
.PP
A source that is itself a timberfs log \(em a rotation segment shipped
from another machine, say \(em is detected automatically (a
.BR .trunk / .rings
path, or a logical name whose pair exists) and its chunks are merged
.B verbatim
into the destination: no decompression, no timestamp parsing, index
included, cost proportional to the compressed size. A segment whose exact
chunks are already in the destination is skipped, so re-running a
shipping script is a no-op; merging refuses to break the destination's
time ordering (import segments oldest-first). Plain and timberfs sources
can be stitched together. Chunk time
windows come from timestamps
.B parsed from the log lines
(historical data's write time is meaningless), so
.B query
works by the times the log actually talks about. Auto-detected timestamps
must sit at the start of the line: RFC 3339 / ISO 8601 and common
variants (space instead of
.BR T ,
dotted dates, and
.BR . / , / :
before the milliseconds \(em logback's
.B yyyy.MM.dd HH:mm:ss:SSS
included) and leading epoch seconds/milliseconds; Apache/CLF
.RB ( [10/Jul/2026:09:23:45\ +0200] )
is the one mid-line exception. Anything else via
.B \-\-timestamp\-regex
(one capture group, searched \(em anchor with
.B ^
if wanted) plus
.B \-\-timestamp\-format
(chrono strftime). Zoneless timestamps are taken as local time unless
.BR \-\-utc .
Lines without a timestamp (stack traces, continuations) inherit the
previous line's, so multiline entries land in the right window; a file
that STARTS mid-entry backfills its head lines with the first timestamp
found, and in a stitch they inherit the previous file's last stamp \(em
exactly right for entries split by rotation. Mildly out-of-order lines
widen chunk windows instead of getting lost. Accepts
.B \-\-chunk\-size
and
.BR \-\-level ,
and
.B \-\-index
declares and builds the
.B .grain
token index \(em needed once per log (the declaration persists in the
.B .bark
manifest; see
.BR create ),
after which every import maintains the index automatically.
.PP
Re-importing is idempotent: a non-empty destination is treated as a
checkpoint. The already-imported bytes are verified against the same
source ranges (every chunk, or first/middle/last with
.BR \-\-quick ),
then only what the source has grown by is appended; an identical source
is a no-op, and a source that shrank or changed is refused before
anything is written (rotated files should be imported to a new target).
.SS export
.B timberfs export
.I SOURCE DEST
copies the chunks overlapping the given window (default: everything) into
a NEW timberfs log \(em verbatim, no recompression, cost proportional to
the compressed size of the window. The source needs no lock and may be
live. A
.I DEST
ending in
.B .timber
writes the single-file transfer bundle instead: a plain uncompressed tar
(the payload is already zstd) holding the
.B .rings
member first and the
.B .trunk
member second.
Bundles are first-class READ-ONLY logs:
.BR query ,
.B index
and
.B export
operate on a
.B .timber
file directly (tar stores its members contiguously and uncompressed, so
the trunk member is simply a trunk at an offset), and
.B import
merges them — unpacking is only ever needed to append. Stock tools always
suffice to recover one:
.B tar xf x.timber && zstd \-dc x.trunk
(a pair tarred by hand follows the same convention). Export always
creates; merging into an existing log is
.BR import 's
job.
.PP
A window that selects nothing still exports: an empty artifact whose bark
records the requested window is evidence of absence (\(lqSaturday was
covered, nothing was there \(em ingest Sunday\(rq), where a MISSING file
is absence of evidence (\(lqa day is missing \(em don't ingest past the
gap\(rq). Opposite signals to a consumer, so an empty result is a result,
not an error;
.B \-\-fail\-on\-empty
makes it one for pipelines that want a quiet day to be loud.
.B import
treats empty sources the same way: skipped with a note, never fatal, and
an all-empty import into a new destination still creates it.
.SS query
.B timberfs query
.I FILE
prints the bytes stamped inside the given time window to standard output,
reading the backing files directly \(em it works with or without an active
mount, but only sees flushed chunks (the still-buffered tail, at most
flush-age old, is visible only through the mount).
.I FILE
is a backing file: the logical name, or its
.B .trunk
or
.B .rings
path.
.TP
.BI \-\-from " TIME"
Start of the window (default: beginning of the file).
.TP
.BI \-\-to " TIME"
End of the window (default: end of the file).
.TP
.BI \-\-has " TOKEN"
Only chunks that (probably) contain
.IR TOKEN ,
tested against the
.B .grain
Bloom index (build it with
.BR "timberfs reindex" ;
without one, \-\-has warns and scans the window). Matching is by WHOLE
token \(em ASCII-alphanumeric runs of 3\(en64 characters, exact case;
substrings do not match. An argument containing separators
.RB ( req\-8f3a )
must match all its tokens in the same chunk; repeat
.B \-\-has
for chunk-level AND. About 1% of non-matching chunks pass anyway (Bloom
false positives) \(em the exact, entry-level filtering belongs in
.B timberfs grep
downstream. The killer use: finding a unique identifier with NO time
bound.
.PP
Several
.I FILE
arguments give the fleet view over per-stream logs: chunks are selected
per file (each file's own rings and grain) and interleaved across files
by their time windows, with within-file order preserved; every output
line carries a grep-style
.RB \(lq path: \(rq
prefix
.RB ( \-\-no\-filename
to suppress). Attribution lives in the filename \(em store one log per
host/app and merge at read time.
.PP
Selection is
.B chunk-granular
by design: every chunk whose write-time window overlaps the requested
range is emitted in full, so the result carries at most flush-age worth of
slop at each edge. The intended workflow is to let
.B query
do the coarse seek into a huge file, then trim exactly with
.BR grep (1)
on the timestamps the log lines carry anyway.
.SS grep
.B timberfs grep
.I PATTERN
.RI [ FILE ]
is an entry-aware grep: a log ENTRY is a line carrying a timestamp plus
all its continuation lines (stack traces, wrapped output), detected with
the same timestamp auto-detection as
.B import
(and the same
.BR \-\-timestamp\-regex / \-\-timestamp\-format
escape hatch). The pattern is matched against the whole entry, and
matching entries print whole; pipe several
.B grep
invocations for entry-level AND. Flags:
.B \-i
(case-insensitive),
.B \-v
(print non-matching entries),
.B \-c
(count matching entries).
.PP
The default matching mode is a
.B literal at token boundaries
(\(lqword mode\(rq: ERROR finds the word ERROR \(em [ERROR], \(lqERROR:\(rq
\(em not ERRORS or PROTOCOLERROR). That is the
.BR .grain 's
own whole-token semantics, so on an indexed log the pattern's tokens
pre-filter chunks
.B automatically and exactly
\(em
.B grep 'FOO BAR ERROR'
is
.B \-\-has FOO \-\-has BAR \-\-has ERROR
at chunk level, then \(lqthe entry contains the anchored phrase\(rq at
entry level; an entry matching the phrase must contain every token
whole, so the skip is conservative, never wrong. (For scattered
word-AND instead of a phrase, use the pattern-less
.B \-\-has
form, or pipe greps.)
.B \-F
matches a raw substring (reaches inside tokens \(em partial ids) and
.B \-\-regex
a full regular expression:
.B both read every chunk, so don't reach for them unless you need them.
.B \-i
(the grain is exact-case) and
.B \-v
(non-matches must be read to be printed \(em an index proving \(lqno
matches here\(rq means print the whole chunk, not skip it) also scan.
Whenever a grain exists but cannot be used, a one-line note on standard
error says why;
.B \-\-scan
forces the full scan.
.PP
Input is a raw log on standard input, a plain log file, or a timberfs
log/bundle \(em in the timberfs case
.BR \-\-from / \-\-to / \-\-has
pre-select chunks first (time index and .grain Bloom filters), the
selection is padded one chunk each way so entries straddling a boundary
arrive whole, and the pattern then matches exactly:
.PP
.nf
.RS
timberfs grep req\-8f3a backing/app.log \-\-has req\-8f3a
cat any.log | timberfs grep 'tenantId=FOO' | timberfs grep \-v DEBUG
timberfs grep req\-8f3a collector/*.log \-\-has req\-8f3a
.RE
.fi
.PP
Several files process in argument order with grep-style
.RB \(lq path: \(rq
line prefixes
.RB ( \-\-no\-filename
to suppress) and per-file counts under
.BR \-c .
.PP
The
.I PATTERN
may be left out when
.BR \-\-has / \-\-from / \-\-to
select instead:
.B timberfs grep \-\-has ERROR \-c backing/app.log
counts the entries containing ERROR \(em the first argument, naming an
existing timberfs log, is taken as the file, and entries containing
every
.B \-\-has
token match (with a pure
.BR \-\-from / \-\-to
window, every entry in it). A note on standard error says what was
matched. (Without this, the file would silently become the PATTERN \(em
grep's first positional \(em and the resulting errors blamed stdin.)
.PP
.B \-\-into
.I DEST
writes the matching entries into a NEW timberfs log (or
.B .timber
bundle) instead of printing them \(em the investigation shipped as an
artifact, the third derivation after
.B export
and rotation. It takes exactly one timberfs source (lineage stays
unambiguous); the artifact's bark records
.B derived_from
plus the operation, fully told:
.B command
(the invocation as typed \(em what question produced this),
.B pattern
and the requested window. Multiline entries stay glued, chunk windows
come from the entries' own timestamps, and the result is a normal store:
queryable, indexable, exportable, importable. Unlike
.B export
(verbatim chunk copies), entries are decompressed, matched and
recompressed \(em cost follows the uncompressed window. A pattern that
matches nothing yields an (empty) artifact attesting exactly that;
.B \-\-fail\-on\-empty
errors instead.
.PP
.nf
.RS
timberfs grep 'tenantId=FOO' backing/app.log \-\-from 13:00 \-\-to 14:10 \-\-into case.timber
timberfs query case.timber | less # attach case.timber to the ticket
.RE
.fi
.SS index
.B timberfs index
.I FILE
prints the chunk index of a backing file: per chunk the uncompressed
offset and length, compressed length, compression ratio, and first/last
write time, plus totals.
.SS reindex
.B timberfs reindex
.I FILE
builds (or rebuilds) the
.B .grain
token index for a log: one Bloom filter per chunk over every distinct
token in it, ~10 bits per token, ~1% false positives \(em roughly 1\(en2%
of the raw data size. Derived data under the sidecar contract: safe to
delete, cheap to rebuild, and rotation/retention drop it (rebuild
afterwards). A grain that lags a growing log is still correct \(em
uncovered chunks are simply scanned.
.SS rotate
.B timberfs rotate \-\-cutoff
.I TIME SOURCE
.RI [ DEST ]
moves every chunk written entirely before
.I TIME
out of
.I SOURCE
and appends it to
.I DEST
(created if missing, must be in the same backing directory). Compressed
frames are relocated verbatim \(em nothing is recompressed \(em so the
cost is proportional to the compressed size. A chunk straddling the cutoff
stays in the source. Appending to an existing
.I DEST
is refused if it would break the destination's time ordering.
.PP
Rotation auto-detects a live mount: if a daemon serves the backing
directory (advertised via the lock file), the request is routed through it
and performed atomically; otherwise the backing files are rewritten
directly under the same lock.
.PP
Rotating nothing into a new
.I DEST
still creates it, empty \(em the same present-but-empty vs missing
distinction as an empty
.BR export ,
for pipelines that rotate-then-ship.
.TP
.B \-\-delete
Drop the rotated chunks instead of moving them (retention). Mutually
exclusive with
.IR DEST .
.TP
.B \-\-dry\-run
Show what would move without changing anything.
.TP
.B \-\-fail\-on\-empty
Error when nothing rotates instead of attesting the empty result.
.SH TIME FORMATS
.I TIME
arguments accept RFC 3339
.RB ( 2026\-07\-10T13:42:00+02:00 ),
.BR "YYYY\-MM\-DD HH:MM" [ :SS ]
(local time; a
.B T
separator also works, and dots as date separators \(em paste straight
from logback-style logs), a bare
.B YYYY\-MM\-DD
(midnight local time, so
.B \-\-from 2026\-07\-10 \-\-to 2026\-07\-11
selects exactly that day),
.BR HH:MM [ :SS ]
(today, local time), and unix epoch seconds or milliseconds.
.SH ON-DISK FORMAT
Each logical file
.I name
is backed by two files in the backing directory:
.TP
.IB name .trunk
The data: a plain concatenation of zstd frames, one per chunk, with no
wrapper bytes. The full uncompressed content is therefore always
recoverable with stock tools:
.B zstd \-dc
.IB name .trunk
.TP
.IB name .rings
The index: an 8-byte magic
.RB \(lq RING0001 \(rq
followed by fixed-size 48-byte records (little-endian u64 fields:
uncompressed start/length, compressed start/length, first/last write time
in unix milliseconds), appended in write order.
.PP
Chunks are written data-first, index-second; on open, index records
pointing past the end of the data are dropped and orphaned data bytes are
overwritten.
.BR fsync (2)
through the mount flushes the buffer as a chunk and syncs both backing
files.
.PP
The daemon holds an exclusive
.BR flock (2)
on
.IB backingdir /.timberfs.lock
recording its mountpoint; offline rotation takes the same lock.
.SH EXTENDED ATTRIBUTES
Files in the mount expose read-only metadata via
.BR getfattr (1):
.BR user.timberfs.chunks ,
.BR user.timberfs.compressed_size ,
.BR user.timberfs.first_write ,
.BR user.timberfs.last_write .
The attribute
.B user.timberfs.rotate
is a write-only control interface used internally by
.BR "timberfs rotate" .
.SH SYSTEMD
The Debian package ships a template unit
.BR timberfs@.service :
create
.IB /etc/timberfs/ instance .conf
defining
.BR BACKING ,
.B MOUNTPOINT
and optionally
.BR EXTRA_OPTS ,
then
.B systemctl enable \-\-now
.BI timberfs@ instance .
Stopping the unit unmounts first, so the daemon flushes all buffers and
exits cleanly. An example configuration is installed under
.IR /usr/share/doc/timberfs/examples/ .
.SH EXAMPLES
Mount and use:
.PP
.nf
.RS
timberfs mount ./logs\-backing ./logs &
myapp >> logs/app.log
tail \-f logs/app.log
.RE
.fi
.PP
Or skip FUSE entirely and pipe:
.PP
.nf
.RS
myapp 2>&1 | timberfs append logs\-backing/app.log
.RE
.fi
.PP
Extract a time window from a huge log, then trim with grep:
.PP
.nf
.RS
timberfs query logs\-backing/app.log \-\-from 13:42 \-\-to 13:43 | grep ERROR
.RE
.fi
.PP
Daily rotation and 30-day retention, no recompression:
.PP
.nf
.RS
timberfs rotate logs\-backing/app.log app\-$(date \-d yesterday +%F).log \\
\-\-cutoff "$(date +%F) 00:00"
timberfs rotate logs\-backing/app.log \-\-delete \\
\-\-cutoff "$(date \-d '30 days ago' +%F) 00:00"
.RE
.fi
.PP
Disaster recovery with stock tools only:
.PP
.nf
.RS
zstd \-dc logs\-backing/app.log.trunk > app.log.recovered
.RE
.fi
.SH EXIT STATUS
0 on success, non-zero on any error (2 for command-line usage errors).
.SH SEE ALSO
.BR fusermount3 (1),
.BR zstd (1),
.BR getfattr (1),
.BR systemd.unit (5)
.PP
Project page and full design notes:
.UR https://github.com/torstei/timberfs
.UE