# Saddle Log File Plan
## Status and scope
This document defines the Observability-owned file layout that follows the V1
structured Log and Trace model. It deliberately does not define a log storage
platform, remote transport, Metrics, alerting, or dashboards.
The current implementation writes every record to one injected `Write`
destination. The target design replaces that single physical stream with a
small, fixed set of streams whose routing is owned by Observability. Applications
must not choose a file per event, service, tenant, trace, or module.
File mode uses bounded asynchronous backpressure. It never performs file I/O on
an application task, but it also never treats a dropped Access, Trace, Event, or
System record as a successful emission. Sampling and drop-on-full are not
enterprise defaults.
## Fixed streams
Each record is written to exactly one stream. All streams use newline-delimited
JSON and retain the existing trace and span correlation fields.
| Access | `saddle-access.jsonl` | `framework.call.*` with `kind=external_request` | Request rate, latency, outcome, and entry-point analysis |
| Trace | `saddle-trace.jsonl` | `framework.call.*` with `kind=service`, `database`, or `transaction` | In-process call graph and dependency analysis |
| Event | `saddle-event.jsonl` | `domain.event` | Domain-event collection and business correlation |
| System | `saddle-system.jsonl` | Observability/runtime lifecycle, degradation, and internal diagnostic records | Runtime health and collection-pipeline diagnosis |
Four is a fixed upper bound for V1, not an application setting. In particular:
- errors stay in the stream of the operation that produced them;
- start and finish records for one call stay in the same stream;
- stream assignment uses typed record metadata, never string-prefix matching at
the file writer;
- no stream is split by log level, service, database, tenant, or trace ID;
- records are never duplicated into an aggregate "all" file.
The System stream is intentionally reserved even if an initial implementation
emits few records to it. Output failure cannot reliably be reported back into
the failed stream, so a short, rate-limited emergency diagnostic may also be
written to stderr. Business payloads must never be copied to that fallback.
## Application configuration
File mode requires one application-supplied value:
```text
log_directory
```
The application chooses the writable root because only the deployment knows its
volume layout. Observability owns all other behavior:
- the four file names and routing rules;
- JSON Lines encoding and schema version;
- file creation and append behavior;
- rotation thresholds and retention;
- flushing, shutdown, and failure reporting.
Application identity is already carried by `CallContext` and must not be
duplicated in logger configuration. Queue sizes, per-stream file names, rotation
thresholds, and retention counts are not application-facing knobs in V1.
The public shape should therefore be a file-specific initializer taking a path.
Stdout may remain an explicit deployment option, but it uses the same typed
streams, bounded asynchronous admission, failure wakeup, and sticky health state
as file mode. Its physical output is multiplexed and every record carries its
logical stream; it is not a weaker synchronous fallback.
Arbitrary writer injection is not a supported deployment contract. A private
test-only writer factory exercises the same queue, admission, health, flush, and
shutdown state machine as production writers. It cannot be selected by business
applications or preserve the current drop-new behavior.
## Directory ownership and recovery
The configured directory is an exclusive Observability resource, not a shared
append target. File-mode initialization must acquire and hold an operating-system
advisory exclusive lock on a well-known lock file in that directory for the
entire observer lifetime. Initialization fails with a stable
`directory-in-use` error if another process owns the lock, and fails as
unsupported if the filesystem cannot provide the required lock semantics.
Process termination releases the operating-system lock; PID inspection or
blind deletion of a stale-looking lock file is never used as an ownership test.
Rotation uses same-directory, no-replace filesystem operations. A rotated name
contains the stream, timestamp, and a newly generated collision-resistant
suffix. The operation must fail or choose another name if any destination entry
already exists; it must never overwrite an existing file. The active name is
published only after the old contents have an immutable rotated name.
After acquiring the directory lock, startup reconciles an interrupted rotation
before accepting records:
- an intact active file is reopened in append mode;
- a completed immutable rotated file is retained as-is;
- an unambiguously staged old active file is promoted to a fresh, non-colliding
rotated name;
- a missing active file is created with create-new semantics;
- ambiguous aliases, unsupported file types, or evidence that two files may
claim the same generation cause initialization to fail rather than delete or
overwrite data.
Retention only considers immutable files matching Saddle's exact generated-name
grammar. It never follows symlinks and never removes the active file, lock file,
unknown entries, or ambiguous recovery artifacts.
## Autonomous file lifecycle
Initial defaults:
- rotate an active file when it reaches 128 MiB;
- retain at most 8 rotated files per stream;
- also remove rotated files older than 7 days;
- apply both limits, deleting the oldest eligible files first;
- generate collision-resistant rotated names and use no-replace publication;
- reopen active files in append mode after process restart;
- create the configured directory if it does not exist.
The maximum retained rotated data is approximately 4 GiB across four streams,
plus active files. These are framework defaults and can evolve with a versioned
Observability release rather than becoming per-application tuning.
Initialization is atomic from the caller's perspective: if any required stream
cannot be opened, file-mode initialization fails and does not silently fall back
to a combined file or stdout. After initialization, healthy writers continue
draining already accepted records if one stream fails, but any unrecoverable
stream, rotation, retention, sync, or disk-capacity failure becomes a sticky
Observer health failure. It is visible immediately through admission and health
interfaces, not deferred until final shutdown. New external requests are
rejected once the Observer is unhealthy.
Each stream needs an independent writer path so a slow or failed destination
does not create head-of-line blocking across unrelated record classes.
## Bounded asynchronous backpressure
Each of the four streams owns a bounded queue and independent capacity permits.
Acquiring capacity is asynchronous; it may suspend the calling task but never
runs or blocks on file I/O. A successful admission means only that the record's
memory capacity is reserved, not that the record has reached the kernel or
durable storage.
Framework calls reserve the complete balanced pair before beginning work:
- starting an external call asynchronously reserves two Access permits for its
start and finish records;
- starting a Service, Database, or Transaction child call asynchronously
reserves two Trace permits;
- the active-call guard owns the finish permit, allowing success, failure,
cancellation, and `Drop` abandonment to enqueue the finish record without
waiting;
- recording a domain event asynchronously acquires one Event permit;
- System producers acquire System capacity before committing the lifecycle
action represented by the record.
This reservation bounds both queued records and in-flight calls while preserving
balanced traces. No successful framework admission can later lose its finish
record merely because the queue became full.
The System queue withholds three permits from general producers for
`shutdown.started`, `admission.stopped`, and `shutdown.finalizing`. The Observer
owns these control permits from initialization. Starting shutdown first consumes
the three reserved permits, materializes the first two bounded control records,
and retains the final permit before closing general capacity acquisition. Normal
System traffic cannot consume the control reserve, so a full System queue cannot
prevent shutdown from starting.
`shutdown.finalizing` is materialized from the retained permit after request
drain and all non-System component results are known. Its fields describe only
those known results and explicitly identify the remaining
`observability_output_finalize` stage. It is delivered before the System
writer's final durability barrier and join, so it never claims that Observer or
application shutdown has completed successfully. Final success is represented
only by the shutdown return value and may be confirmed by an out-of-process
supervisor; no record written through the writer being closed can prove its own
successful final synchronization and join.
If a writer terminates or enters sticky failure, all capacity waiters are woken
with a stable error. They never wait indefinitely. The external-entry adapter
uses that result to stop admission and reject new requests. After materializing
its control records, shutdown stops new capacity acquisition, wakes unadmitted
waiters with a shutdown error, and allows already admitted requests to converge
using their reserved permits. It then drains all accepted records, applies the
durability barrier, and joins the writers.
If a writer has already failed, reserved call and System control records still
materialize in bounded memory and release their permits deterministically. The
sticky error aggregates every undeliverable stream and stage into shutdown's
result; shutdown never waits for failed capacity to recover. The framework never
reports observability success for an undeliverable record.
All supported outputs share this contract. Sampling, priority dropping, silent
fallback, and a synchronous drop-new constructor are not part of the public
Observer API.
## Durability and visible failure
File mode defines these states:
- **admitted**: bounded memory is reserved and the record can be enqueued;
- **written**: the complete JSON line has been accepted by the file write
operation, normally into the operating-system page cache;
- **durable**: all earlier records for the stream are covered by a successful
file data synchronization barrier.
The first write after a successful synchronization starts a one-second deadline
owned by that stream's writer. The deadline is independent of later traffic:
becoming idle does not cancel it, and the writer performs the barrier no later
than the deadline even if no additional command arrives. Reaching 4 MiB since
the last barrier triggers it earlier. A barrier failure becomes sticky health
immediately and wakes admission waiters.
`flush` is a cross-stream barrier: it waits for all records admitted before the
call, flushes userspace buffers, performs file data synchronization on every
stream, and succeeds only if they are durable. Order between different streams
is not promised. `shutdown` stops admission after reserving its System control
records, drains admitted records, performs the same durability barrier, and
joins every writer.
A sudden process crash may lose records still admitted or queued but not
written. A host or power failure may additionally lose written records since the
last successful synchronization barrier. Therefore the bounded loss window is
at most each stream's bounded admitted backlog plus the smaller of one second or
4 MiB of already written data. A successful `flush` or `shutdown` closes that
window for all records preceding its barrier, subject to the storage device's
truthful synchronization guarantees.
Serialization, write, newline, rotation, synchronization, disk-full, and
retention failures are sticky and identify the stream and output stage.
Retention deletion failure is not ignored: it marks the Observer unhealthy,
wakes admission waiters, rejects new external requests, and remains visible from
health, `flush`, and `shutdown`. A short rate-limited stderr diagnostic without
business payload may announce the failure, but stderr is never a record
fallback.
## Stable collection contract
Collectors may rely on:
- stable active file names;
- one JSON record per line;
- exactly one logical stream per record;
- `timestamp_ms`, `level`, `event`, `trace_id`, `span_id`, and
`parent_span_id` retaining their current meanings;
- call outcome and duration remaining on call-finish records;
- rotated files being immutable.
Collectors must discover rotated files by the stable stream prefix rather than
parsing an application-selected filename.
Future schema changes must add an explicit schema version before changing field
meaning or removing fields. File routing must remain based on record type and
call kind so event-name evolution does not silently move data between streams.
## Implementation stages
1. Introduce an internal typed stream classification and test the routing table.
2. Implement directory locking, interrupted-rotation recovery, and the four-file
no-replace rotating writer behind a single-directory initializer.
3. Implement per-stream asynchronous capacity reservation, balanced call
permits, failure wakeup, health propagation, and shutdown convergence.
4. Add deterministic durability-barrier, sticky-failure, and bounded-backpressure
tests, including one write followed by complete inactivity past the sync
deadline and a synchronization failure at that deadline.
5. Add integration tests for same-directory dual initialization, restart name
collision, interrupted rotation recovery, immutable no-overwrite publication,
retention failure, disk-full/write failure, System queue saturation when
shutdown starts, failed-System-writer shutdown aggregation, and a successfully
written `shutdown.finalizing` record followed by final durability-barrier
failure. The latter must keep the phase name and pending-stage field truthful
while shutdown returns the final synchronization error. Also verify trace
correlation across Access and Trace files.
6. Document collector glob patterns and migration from the current stdout
stream.