timberfs 0.10.1

.TH TIMBERFS 1 "2026-07-10" "timberfs 0.1.0" "User Commands"
.SH NAME
timberfs \- append-only, transparently compressed, write-time-indexed filesystem for log files
.SH SYNOPSIS
.B timberfs mount
.RI [ options ]
.I BACKING
.I MOUNTPOINT
.br
.B timberfs create
.RB [ \-\-index ]
.RB [ \-\-set
.IR KEY=VALUE ]...
.I DEST
.br
.B timberfs append
.RI [ options ]
.B \-\-into
.I DEST
.br
.B timberfs import
.RI [ options ]
.B \-\-into
.I DEST
.IR SOURCE .\|.\|.
.br
.B timberfs export
.I SOURCE
.B \-\-into
.I DEST
.RB [ \-\-from
.IR TIME ]
.RB [ \-\-to
.IR TIME ]
.I SOURCE
.I DEST
.br
.B timberfs query
.RB [ \-\-from
.IR TIME ]
.RB [ \-\-to
.IR TIME ]
.IR FILE .\|.\|.
.br
.B timberfs index
.I FILE
.br
.B timberfs reindex
.I FILE
.br
.B timberfs rotate
.B \-\-cutoff
.I TIME
.I SOURCE
.RI [ DEST ]
.RB [ \-\-delete ]
.RB [ \-\-dry\-run ]
.SH DESCRIPTION
.B timberfs
is a FUSE filesystem purpose-built for log files. Files in the mount look
and behave like ordinary log files \(em loggers append,
.BR tail (1)
and
.BR grep (1)
work unmodified \(em but the data is stored in a backing directory as
chunks compressed with zstd, together with a per-chunk
.B write-time index
that records when each chunk was written. Asking \(lqwhat was written
between 13:42 and 13:43?\(rq is a binary search plus a few frame
decompressions, independent of file size.
.PP
Files are append-only: writes anywhere but the end of the file fail with
.BR EPERM .
Truncating a file to zero is allowed and starts it over
(copytruncate-style rotation); rename and unlink work normally.
.B ls \-l
shows the logical (uncompressed) size while
.B du
shows the real, compressed disk usage.
.SH SUBCOMMANDS
.SS mount
.B timberfs mount
.I BACKING MOUNTPOINT
serves the logical view of backing directory
.I BACKING
on
.IR MOUNTPOINT ,
in the foreground. Unmount with
.B fusermount3 \-u
.IR MOUNTPOINT .
.TP
.BI \-\-chunk\-size " BYTES"
Uncompressed buffer size that triggers a chunk flush (default 262144).
.TP
.BI \-\-level " N"
zstd compression level (default 3).
.TP
.BI \-\-flush\-age " SECONDS"
Maximum time appended data may sit unflushed (default 5). Bounds both the
time granularity of the index and the data lost in a crash without
.BR fsync (2).
.TP
.B \-\-allow\-other
Let other users access the mount. Unprivileged users need
.B user_allow_other
in
.IR /etc/fuse.conf ;
root (e.g. the systemd unit) does not.
.SS create
.B timberfs create
.I DEST
makes an empty log with its properties declared up front in a
.B .bark
manifest \(em one flat, optional, human-editable JSON object beside the
pair.
.B \-\-index
declares the token index, database-style: once declared, EVERY import
maintains the
.B .grain
automatically (extended incrementally for new chunks, rebuilt if it went
missing \(em say, dropped by rotation or retention), so there is no
per-import flag to forget.
.B \-\-set
.I KEY=VALUE
records free-form provenance
.RB ( host ,
.BR path ,
.BR format ,
\|.\|.\|.). Every manifest is minted with a durable identity on first write:
.B id
(a random UUID, constant across renames, moves and hosts \(em the
identity of the store, where paths are merely its current address) and
.B created
(RFC3339). Artifacts made by
.B export
and by rotation into a new segment are NEW stores: they get a fresh
identity, a
.B derived_from
pointer to the source's id (lineage chains compose across re-carves and
shipping), a
.B derived_op
of
.BR export / rotate
(copy vs move \(em whether the source still holds the data), and inherit
the data-provenance fields; settings
.RB ( index )
do not inherit. Rotation mints its source's identity when missing (it
holds the writer locks); export never writes its source. Content facts
\(em actual spans, sizes \(em are deliberately NOT recorded in bark: the
artifact's own rings state them authoritatively. The REQUESTED window is
different \(em an operation fact, like
.B derived_op
\(em and a windowed export records it as
.BR window_from / window_to
(RFC3339 UTC; only the bounds given; never inherited). Content can never
state coverage \(em an artifact whose last line is 17:00 doesn't say
whether 17:00\(en24:00 was covered-but-silent or not covered \(em and for
an EMPTY artifact the declared window is its entire meaning. Unlike the
derived
.BR .grain ,
bark survives head-drops, travels on rename, and ships inside
.B .timber
bundles.
.SS set
.B timberfs set
.I FILE KEY=VALUE\|.\|.\|.
declares or changes a store's properties in its
.B .bark
manifest \(em validated and atomic, which hand-editing the JSON is
neither. Known settings are parse-checked
.RB ( retain =90d,
.BR retain_size =50G,
.BR index =true|false);
anything else is free-form provenance
.RB ( host =...).
.B \-\-unset
.I KEY
removes a key. Identity and lineage
.RB ( id ,
.BR created ,
.BR derived_* )
are facts, not settings, and are refused. Works on LIVE stores: writers
re-read the manifest on their once-a-second retention tick, so
.B timberfs set backing/app.log retain=30d
takes effect within a second with no restart \(em restarting a writer
means restarting whatever pipes into it. A manifest that stops parsing
mid-flight (a bad hand-edit) keeps the writer alive on its last good
policy, with one warning: never silently unbounded, never a dead
producer.
.SS append
.B timberfs append \-\-into
.I DEST
reads standard input and appends it to a log in the backing store directly,
with no FUSE mount involved \(em the
.BR svlogd (8)/s6-log
pattern (destinations are always named
.BR \-\-into ,
here as everywhere):
.PP
.nf
.RS
myapp 2>&1 | timberfs append \-\-into logs\-backing/app.log
.RE
.fi
.PP
With
.B \-\-records
standard input is a
.BR timberfs\-records (5)
stream instead of raw text: entries arrive pre-framed, and an entry
carrying its original write window
.RB ( wf / wl )
keeps it \(em write history survives the pipe. Without a carried
window, append stamps now, as always \(em the stream's word is law,
silence falls back to the command's own clock. Delivery is streaming:
data lands as it arrives, and a truncated stream keeps what came and
fails the exit code (at-least-once \(em re-running may duplicate; use
.B import \-\-records
for all-or-nothing).
.PP
Each log has exactly one writer (an exclusive per-file lock), but
appenders for different files happily share a backing directory; a
directory served by a
.B timberfs mount
is unavailable to appenders and vice versa. Appending to an existing log
continues it. Data is flushed into chunks by the same size and age rules
as the mount; end of input,
.B SIGTERM
or
.B SIGINT
flush and sync everything before exit. Accepts the same
.BR \-\-chunk\-size ,
.B \-\-level
and
.B \-\-flush\-age
options as
.BR mount ,
plus:
.TP
.BI \-\-retain " DURATION"
Continuously drop data older than
.I DURATION
(e.g.
.BR 30d ,
.BR 12h ,
.BR 90m ).
.TP
.BI \-\-retain\-size " SIZE"
Keep the on-disk (compressed) size of the log at or under
.I SIZE
(e.g.
.BR 200G ,
.BR 512M ;
powers of 1024), dropping the oldest data first. Combines with
.BR \-\-retain :
whichever limit bites first wins.
.PP
Retention is checked every second, at startup and once more at shutdown.
Dropping the head currently compacts the log by rewriting the remaining
data, so enforcement is batched: age-expired data is dropped once it makes
up about a tenth of the file, and a size overrun is trimmed down to 95% of
the budget. Compaction briefly needs free disk space proportional to the
retained data.
.SS import
.B timberfs import \-\-into
.I DEST
.IR SOURCE .\|.\|.
converts existing plain log files into a timberfs log. The destination is
a named flag on purpose: a shell glob among the sources can never
swallow it. With
.B \-\-records
the single source (a file, or standard input when none is given) is a
.BR timberfs\-records (5)
stream: entries arrive pre-framed, an entry carrying its original
write window
.RB ( wf / wl )
keeps it \(em replicating a store preserves its write history \(em
and one without falls back to import's own clock, the entry's parsed
timestamp (the backfill doctrine). Delivery is ATOMIC: nothing is
visible until the stream's
.B stream\-end
arrives, and a truncated stream leaves the store byte-for-byte
unchanged. The upstream selection echo and every pipeline
.B stage=
land in the destination's manifest \(em an artifact remembers the
pipe that filled it:
.PP
.nf
.RS
timber\-filter \-\-records \-\-has ERROR \-\-from 13:00 app.log |
    timberfs import \-\-records \-\-into case.timber.d/case.log
timberfs query \-\-records app.log | ssh host timberfs import \-\-records \-\-into mirror/app.log
.RE
.fi Several sources \(em a rotated set \(em may be given
in any order: each file's first parsed timestamp determines the stitch
order (rotation numbering and glob order lie), and the plan is printed
before anything is written. Two sources starting at the same timestamp
are refused as likely duplicates.
.PP
Each plain source is placed by its
.B first timestamp
relative to what the destination already holds. Starting
.I after
the destination's last write, it simply appends (the daily bulk-load).
Starting
.I inside
the destination's window \(em day files cut with slack, an import re-run
\(em the overlap is deduplicated
.BR "line by line" :
lines the destination already has are skipped, genuinely new lines in
the covered window are imported (with a warning, since that usually
means the captures disagreed), and re-importing an already-covered file
is a clean no-op. Counts are reported. Starting
.I before
everything in the destination is refused \(em import in chronological
order or to a new target. A source starting exactly
.B where the destination starts
is treated as the same file, regrown: the already-imported prefix is
byte-verified and only the growth is appended (truncated or rewritten
files are refused before anything is written; see
.BR \-\-quick ).
.PP
A source that is itself a timberfs log \(em a rotation segment shipped
from another machine, say \(em is detected automatically (a
.BR .trunk / .rings
path, or a logical name whose pair exists) and its chunks are merged
.B verbatim
into the destination: no decompression, no timestamp parsing, index
included, cost proportional to the compressed size. A segment whose exact
chunks are already in the destination is skipped, so re-running a
shipping script is a no-op; merging refuses to break the destination's
time ordering (import segments oldest-first). Plain and timberfs sources
can be stitched together. Chunk time
windows come from timestamps
.B parsed from the log lines
(historical data's write time is meaningless), so
.B query
works by the times the log actually talks about. Auto-detected timestamps
must sit at the start of the line: RFC 3339 / ISO 8601 and common
variants (space instead of
.BR T ,
dotted dates, and
.BR . / , / :
before the milliseconds \(em logback's
.B yyyy.MM.dd HH:mm:ss:SSS
included) and leading epoch seconds/milliseconds; Apache/CLF
.RB ( [10/Jul/2026:09:23:45\ +0200] )
is the one mid-line exception. Anything else via
.B \-\-timestamp\-regex
(one capture group, searched \(em anchor with
.B ^
if wanted) plus
.B \-\-timestamp\-format
(chrono strftime). Zoneless timestamps are taken as local time unless
.BR \-\-utc .
Lines without a timestamp (stack traces, continuations) inherit the
previous line's, so multiline entries land in the right window; a file
that STARTS mid-entry backfills its head lines with the first timestamp
found, and in a stitch they inherit the previous file's last stamp \(em
exactly right for entries split by rotation. Mildly out-of-order lines
widen chunk windows instead of getting lost. Accepts
.B \-\-chunk\-size
and
.BR \-\-level ,
and
.B \-\-index
declares and builds the
.B .grain
token index \(em needed once per log (the declaration persists in the
.B .bark
manifest; see
.BR create ),
after which every import maintains the index automatically.
.PP
Re-importing is idempotent: a non-empty destination is treated as a
checkpoint. The already-imported bytes are verified against the same
source ranges (every chunk, or first/middle/last with
.BR \-\-quick ),
then only what the source has grown by is appended; an identical source
is a no-op, and a source that shrank or changed is refused before
anything is written (rotated files should be imported to a new target).
.SS export
.B timberfs export
.I SOURCE
.B \-\-into
.I DEST
copies the chunks overlapping the given window (default: everything) into
a NEW timberfs log \(em verbatim, no recompression, cost proportional to
the compressed size of the window. The source needs no lock and may be
live. A
.I DEST
ending in
.B .timber
writes the single-file transfer bundle instead: a plain uncompressed tar
(the payload is already zstd) holding the
.B .rings
member first and the
.B .trunk
member second.
Bundles are first-class READ-ONLY logs:
.BR query ,
.B index
and
.B export
operate on a
.B .timber
file directly (tar stores its members contiguously and uncompressed, so
the trunk member is simply a trunk at an offset), and
.B import
merges them — unpacking is only ever needed to append. Stock tools always
suffice to recover one:
.B tar xf x.timber && zstd \-dc x.trunk
(a pair tarred by hand follows the same convention). Export always
creates; merging into an existing log is
.BR import 's
job.
.PP
A window that selects nothing still exports: an empty artifact whose bark
records the requested window is evidence of absence (\(lqSaturday was
covered, nothing was there \(em ingest Sunday\(rq), where a MISSING file
is absence of evidence (\(lqa day is missing \(em don't ingest past the
gap\(rq). Opposite signals to a consumer, so an empty result is a result,
not an error;
.B \-\-fail\-on\-empty
makes it one for pipelines that want a quiet day to be loud.
.B import
treats empty sources the same way: skipped with a note, never fatal, and
an all-empty import into a new destination still creates it.
.SS query
.B timberfs query
.I FILE
prints the log entries inside the given time window to standard output,
reading the backing files directly \(em it works with or without an active
mount, but only sees flushed chunks (the still-buffered tail, at most
flush-age old, is visible only through the mount).
.I FILE
is a backing file: the logical name, or its
.B .trunk
or
.B .rings
path.
.TP
.BI \-\-from " TIME"
Start of the window (default: beginning of the file).
.TP
.BI \-\-to " TIME"
End of the window (default: end of the file).
.TP
.BI \-\-has " TOKEN"
Only chunks that (probably) contain
.IR TOKEN ,
tested against the
.B .grain
Bloom index (build it with
.BR "timberfs reindex" ;
without one, \-\-has warns and scans the window). Matching is by WHOLE
token \(em ASCII-alphanumeric runs of 3\(en64 characters, exact case;
substrings do not match. An argument containing separators
.RB ( req\-8f3a )
must match all its tokens in the same chunk; repeat
.B \-\-has
for chunk-level AND;
.BI \-\-any " TEXT"
(repeatable) keeps chunks where at least ONE alternative matches \(em
the union of exact branches, still exact \(em and composes with
.BR \-\-has .
About 1% of non-matching chunks pass anyway (Bloom
false positives) \(em the exact, entry-level filtering belongs in
.BR timber\-filter (1)
downstream. The killer use: finding a unique identifier with NO time
bound.
.TP
.BR \-f ", " \-\-follow
After the selected output, keep emitting entries as chunks are committed,
until interrupted \(em like
.BR tail (1)
.BR \-f .
A flushed chunk is the unit of visibility, so new data appears within the
writer's
.B \-\-flush\-age
(default 5s), not per line. Read-only and lock-free, so it runs beside a live
appender; plain text streams raw bytes, while
.BR \-0 ", " \-\-records ", and " \-\-show\-write\-time
run the entry pipeline. Filter a live follow with a pipe
.RB ( "| grep" );
.B \-\-has
selects whole chunks offline and does not compose with it.
.TP
.BI \-\-tail " N"
Show (about) the last
.I N
log entries \(em a stamped line and its continuation lines are one entry \(em
then, with
.BR \-\-follow ,
keep going. Chunk-granular: rounded out to a whole chunk, so a few extra may
precede the Nth-from-last. (A store with no parseable timestamps counts lines.)
.TP
.BI \-\-max " N"
Stop after at most
.I N
entries (a hard cap, like
.BR head (1)).
Composes with
.B \-\-follow
to bound it; conflicts with
.BR \-\-tail .
.PP
Several
.I FILE
arguments give the fleet view over per-stream logs: chunks are selected
per file (each file's own rings and grain) and interleaved across files
by their time windows, with within-file order preserved; every output
line carries a grep-style
.RB \(lq path: \(rq
prefix
.RB ( \-\-no\-filename
to suppress). Attribution lives in the filename \(em store one log per
host/app and merge at read time.
.PP
Selection is
.B chunk-granular
by design: every chunk whose write-time window overlaps the requested
range is emitted in full, so the result carries at most flush-age worth of
slop at each edge. The intended workflow is to let
.B query
do the coarse seek into a huge file, then trim exactly with
.BR timber\-filter (1)
on the timestamps the log lines carry anyway.
.SS searching
Matching lives in its own tool:
.BR timber\-filter (1),
the entry-aware pipeline filter. A store argument is searched through
the selection layer (it runs
.B timberfs query \-\-records
itself), so the fused and piped spellings are the same search:
.PP
.nf
.RS
timber\-filter \-\-has ERROR \-\-from 13:00 backing/app.log
timberfs query \-\-records \-\-from 13:00 backing/app.log | timber\-filter \-\-has ERROR
.RE
.fi
.PP
The investigation-as-artifact workflow is a pipe into the records sink
\(em the destination's manifest records the selection and every stage
that shaped the data:
.PP
.nf
.RS
timber\-filter \-\-records \-\-has 'tenantId=FOO' \-\-from 13:00 backing/app.log |
    timberfs import \-\-records \-\-into case/case.log
timberfs export case/case.log \-\-into case.timber   # attach to the ticket
.RE
.fi
.SS info
.B timberfs info
.I FILE
shows a store's vital signs on one screen: identity and creation time,
lineage (what it derives from, by which operation, over which requested
window, and \(em for records-sink
artifacts \(em the command line that produced it, labelled
.BR question ),
declared provenance, logical size vs on-disk size and compression
ratio, the time span covered and its duration, index sizes with grain
.B coverage
(a grain lagging its log \(em appender writes, partial extends \(em
covers fewer chunks than the rings; the gap is scanned), and the writer
state (none / active appender or import / mounted, probed from the
actual flocks, never from stale lock-file contents). Works identically
on backing pairs and read-only
.B .timber
bundles.
.B \-\-json
prints the same facts as one JSON object for scripting. Read-only and
safe against live writers.
.SS index
.B timberfs index
.I FILE
prints the chunk index of a backing file: per chunk the uncompressed
offset and length, compressed length, compression ratio, and first/last
write time, plus totals.
.SS list
.B timberfs list
.RI [ DIR " ...]"
is the directory-level complement to
.BR info :
one row per store, sorted by forest then handle, across every configured
.B forest
(see
.B FORESTS
below), or \(em when one or more
.I DIR
arguments are given \(em exactly the stores in those directories
(ad-hoc; they need not be configured forests). Columns:
.B HANDLE
(the forest handle),
.B FOREST
(the config filename minus
.IR .conf ,
or the directory itself for an ad-hoc/
.B TIMBERFS_FORESTS
forest),
.B SIZE
(compressed on-disk size),
.B SPAN
(earliest \(em latest write time covered, or
.IR empty ),
.B WRITER
.RI ( live
if a writer currently holds the file's own lock, else
.IR \- ),
.B INDEX
.RI ( grain
if a
.B .grain
token index is present or declared, else
.IR \- ),
and
.B RETAIN
(declared
.BR retain / retain_size ,
else
.IR \- ).
The same handle found in two forests shows up as two rows \(em
.B list
is how an ambiguity that handle resolution would refuse to guess at
becomes visible, not something it hides.
.TP
.B \-\-names
Bare handles only, one per line, no header or columns \(em what shell
completion consumes.
.TP
.B \-\-json
A JSON array of objects
.RI ( handle ", " forest ", " dir ", " path ", " size_bytes ", "
.IR from_ms ", " to_ms ", " writer_live ", " indexed ", " retain ", "
.IR retain_size )
instead of the human table.
.PP
No forests configured and no
.I DIR
given prints a note to stderr and exits 0; a forest or
.I DIR
that doesn't exist is skipped (also noted). Read-only and lock-free \(em
like
.BR info ,
it only ever probes a writer lock, never acquires one, so it works for a
non-root reader on a root-owned forest.
.SS reindex
.B timberfs reindex
.I FILE
builds (or rebuilds) the
.B .grain
token index for a log: one Bloom filter per chunk over every distinct
token in it, ~10 bits per token, ~1% false positives \(em roughly 1\(en2%
of the raw data size. Derived data under the sidecar contract: safe to
delete, cheap to rebuild, and rotation/retention drop it (rebuild
afterwards). A grain that lags a growing log is still correct \(em
uncovered chunks are simply scanned.
.SS rotate
.B timberfs rotate \-\-cutoff
.I TIME SOURCE
.RI [ DEST ]
moves every chunk written entirely before
.I TIME
out of
.I SOURCE
and appends it to
.I DEST
(created if missing, must be in the same backing directory). Compressed
frames are relocated verbatim \(em nothing is recompressed \(em so the
cost is proportional to the compressed size. A chunk straddling the cutoff
stays in the source. Appending to an existing
.I DEST
is refused if it would break the destination's time ordering.
.PP
Rotation auto-detects a live mount: if a daemon serves the backing
directory (advertised via the lock file), the request is routed through it
and performed atomically; otherwise the backing files are rewritten
directly under the same lock.
.PP
Rotating nothing into a new
.I DEST
still creates it, empty \(em the same present-but-empty vs missing
distinction as an empty
.BR export ,
for pipelines that rotate-then-ship.
.TP
.B \-\-delete
Drop the rotated chunks instead of moving them (retention). Mutually
exclusive with
.IR DEST .
.TP
.B \-\-dry\-run
Show what would move without changing anything.
.TP
.B \-\-fail\-on\-empty
Error when nothing rotates instead of attesting the empty result.
.SH RETENTION
Retention is a property of the LOG, declared in its manifest
.RB ( retain
= keep at least this long,
.B retain_size
= compressed-size budget, oldest dropped first, whichever bites first)
and enforced by
.B every
writer while it holds the writer locks: the appender and the mount
daemon on their once-a-second tick, and
.B import
after each run \(em so timer-mode (cron import) stores and mounted
stores get retention too, with no appender running. Declare it at
.BR create ,
with
.BR set ,
or via the appender's
.BR \-\-retain / \-\-retain\-size
flags (which persist the declaration \(em all roads converge, and a
restarted writer can no longer silently lose its policy). Absent keys
mean no limits: an artifact carved by
.B grep \-\-into
has no business expiring, which is also why retention, like all
settings, is never inherited by derived stores.
.B timberfs info
shows the declared policy and calls out a store that is over budget
with no writer running to enforce it.
.SH FORESTS
A
.I forest
is a directory
.B timberfs
searches for stores by a short
.IR handle ,
so
.B timberfs query nginx
finds
.I /var/log/timberfs/nginx/nginx.log
without spelling out the path. Handle lookup applies to the source
argument of
.BR query ", " info ", " index ", " reindex ", " set ", " rotate " and " export ;
a full path, a relative path or a
.B .timber
bundle is always taken literally, so every existing invocation is
unchanged \(em a forest is consulted only for a bare token (no
.BR / )
that names no store on disk.
.PP
Forests are configured by
.IR /etc/timberfs/forests.d/*.conf ,
one forest per file, the same
.B KEY=VALUE
idiom as an instance
.IR .conf ,
read in sorted filename order. The only key is
.BR DIR ,
an absolute directory; blank lines,
.B #
comments and unknown keys are ignored, and a configured directory that
does not exist is skipped. The package ships
.I default.conf
with
.BR DIR=/var/log/timberfs ;
edit it, drop in another
.IR .conf ,
or delete it to disable handle lookup (it is a conffile, so edits
survive upgrades).
.PP
A store's handle is its
.B .rings
file name minus
.B .rings
and a single trailing
.BR .log ,
so a flat
.I nginx.rings
and a nested
.I nginx/nginx.log.rings
both resolve as
.BR nginx ", while " metrics.jsonl.rings " keeps its " .jsonl .
Each forest's root and its immediate subdirectories are scanned. A
handle that matches no store, or more than one, is an error that names
what was searched and asks for a full path.
.PP
The environment variable
.B TIMBERFS_FORESTS
(colon-separated absolute directories) replaces the configuration
entirely \(em for tests and one-off use.
.SH SHELL COMPLETION
The package ships completion scripts for
.B bash
and
.BR zsh ,
installed to the standard vendor locations
.RI ( /usr/share/bash\-completion/completions/timberfs " and "
.IR /usr/share/zsh/vendor\-completions/_timberfs )
so both shells load them automatically, with no per-user setup.
.B timberfs
.RB [ TAB ]
lists the subcommands; the store argument of
.BR query ", " info ", " index ", " reindex ", " set " and the source of "
.BR rotate " and " export
additionally completes bare
.B FORESTS
handles, gathered live from
.B timberfs list \-\-names
and offered alongside normal file-path completion. With no forests
configured (or if that call errors), completion silently falls back to
file paths \(em never an error at the prompt.
.SH GLOBAL OPTIONS
.TP
.B \-\-quiet
Suppress informational notes on standard error \(em scan reports,
progress, stitch plans, summaries. Errors and warnings still print.
Accepted by every subcommand. (Long-only on purpose:
.B \-q
keeps its grep(1) meaning available.)
.SH THE TWO CLOCKS
Every entry effectively has two timestamps: the one its line CARRIES
(what happened when \(em the one you can see) and the WRITE time it
arrived in the store (the one the index searches). For imported data
they are identical; for live data they differ by buffering. By default,
.B query
and
.B grep
select chunks by the write-time index (widened by about a minute to
catch buffered stragglers), then verify every entry against
.BR \-\-from / \-\-to
by the timestamp its own line carries \(em so the output answers the
question you asked, in the timestamps you can see. Entries whose
timestamps cannot be read are always included, never hidden; a store
with no parseable timestamps at all falls back to the raw, unwidened
write-time window with a note.
.B \-\-show\-write\-time
annotates each entry with the write window it arrived in \(em and, when
its own stamp falls outside that window, the offset \(em making any
divergence visible and explainable.
.B \-\-by\-write\-time
is the raw escape hatch: chunk output selected by write time only, no
parsing (the pre-0.7.4 behavior).
.B \-0
.RB ( \-\-null )
emits NUL-terminated entry RECORDS \(em a multiline entry (stack trace)
stays one record \(em for
.BR "xargs \-0" ,
.BR "sort \-z" ,
.B "uniq \-z"
pipelines.
.B \-\-records
emits the typed record stream for timber-aware consumers: the same
NUL-framed entries interleaved with metadata records carrying the
format version, an echo of the selection, per-source index statistics,
each entry's length, own timestamp and write window, and end-of-stream
totals whose presence proves the stream arrived complete \(em see
.BR timberfs\-records (5)
for the grammar. Plain
.B \-0
stays the choice for ordinary Unix tools. Persistent whole-hour offsets between the two clocks are
detected and warned about once (on one host the clocks cancel, so such
an offset is a timezone or format misconfiguration, not clock skew).
Declare exotic line formats once in the manifest
.RB ( timestamp_regex ,
.BR timestamp_format ,
.B timestamp_utc
via
.B timberfs set
\(em or import's flags, which persist the declaration); they describe
the CONTENT, so unlike settings they inherit into derived artifacts.
Do not backfill historical data through
.B append
(its index gets today's write times): import the file instead.
.SH TIME FORMATS
.I TIME
arguments accept RFC 3339
.RB ( 2026\-07\-10T13:42:00+02:00 ),
.BR "YYYY\-MM\-DD HH:MM" [ :SS ]
(local time; a
.B T
separator also works, and dots as date separators \(em paste straight
from logback-style logs), a bare
.B YYYY\-MM\-DD
(midnight local time, so
.B \-\-from 2026\-07\-10 \-\-to 2026\-07\-11
selects exactly that day),
.BR HH:MM [ :SS ]
(today, local time), and unix epoch seconds or milliseconds.
.SH ON-DISK FORMAT
Each logical file
.I name
is backed by two files in the backing directory:
.TP
.IB name .trunk
The data: a plain concatenation of zstd frames, one per chunk, with no
wrapper bytes. The full uncompressed content is therefore always
recoverable with stock tools:
.B zstd \-dc
.IB name .trunk
.TP
.IB name .rings
The index: an 8-byte magic
.RB \(lq RING0001 \(rq
followed by fixed-size 48-byte records (little-endian u64 fields:
uncompressed start/length, compressed start/length, first/last write time
in unix milliseconds), appended in write order.
.PP
Chunks are written data-first, index-second; on open, index records
pointing past the end of the data are dropped and orphaned data bytes are
overwritten.
.BR fsync (2)
through the mount flushes the buffer as a chunk and syncs both backing
files.
.PP
The daemon holds an exclusive
.BR flock (2)
on
.IB backingdir /.timberfs.lock
recording its mountpoint; offline rotation takes the same lock.
.SH EXTENDED ATTRIBUTES
Files in the mount expose read-only metadata via
.BR getfattr (1):
.BR user.timberfs.chunks ,
.BR user.timberfs.compressed_size ,
.BR user.timberfs.first_write ,
.BR user.timberfs.last_write .
The attribute
.B user.timberfs.rotate
is a write-only control interface used internally by
.BR "timberfs rotate" .
.SH SYSTEMD
The Debian package ships a template unit
.BR timberfs@.service :
create
.IB /etc/timberfs/ instance .conf
defining
.BR BACKING ,
.B MOUNTPOINT
and optionally
.BR EXTRA_OPTS ,
then
.B systemctl enable \-\-now
.BI timberfs@ instance .
Stopping the unit unmounts first, so the daemon flushes all buffers and
exits cleanly. An example configuration is installed under
.IR /usr/share/doc/timberfs/examples/ .
.PP
A socket\-activated log\-intake pair,
.B timberfs\-log@.socket
and
.BR timberfs\-log@.service ,
streams a producer's
.BR timberfs\-records (5)
stream into a store over a FIFO.
The full directory layout, both unit families, the ownership model and
self\-restart on upgrade are documented in
.IR /usr/share/doc/timberfs/deployment.md .
.SH EXAMPLES
Mount and use:
.PP
.nf
.RS
timberfs mount ./logs\-backing ./logs &
myapp >> logs/app.log
tail \-f logs/app.log
.RE
.fi
.PP
Or skip FUSE entirely and pipe:
.PP
.nf
.RS
myapp 2>&1 | timberfs append \-\-into logs\-backing/app.log
.RE
.fi
.PP
Extract a time window from a huge log, then trim with grep:
.PP
.nf
.RS
timberfs query logs\-backing/app.log \-\-from 13:42 \-\-to 13:43 | grep ERROR
.RE
.fi
.PP
Daily rotation and 30-day retention, no recompression:
.PP
.nf
.RS
timberfs rotate logs\-backing/app.log app\-$(date \-d yesterday +%F).log \\
    \-\-cutoff "$(date +%F) 00:00"
timberfs rotate logs\-backing/app.log \-\-delete \\
    \-\-cutoff "$(date \-d '30 days ago' +%F) 00:00"
.RE
.fi
.PP
Disaster recovery with stock tools only:
.PP
.nf
.RS
zstd \-dc logs\-backing/app.log.trunk > app.log.recovered
.RE
.fi
.SH EXIT STATUS
0 on success, non-zero on any error (2 for command-line usage errors).
.SH SEE ALSO
.BR timberfs-records (5)
.br
.BR fusermount3 (1),
.BR zstd (1),
.BR getfattr (1),
.BR systemd.unit (5)
.PP
Project page and full design notes:
.UR https://github.com/torstei/timberfs
.UE